hf2q 0.1.1

//! Engine — model-owning worker thread with serialized FIFO dispatch.
//!
//! ADR-005 Phase 2 Decision #2: the inference engine runs one request at a
//! time under a serialized FIFO queue. This module implements that model via
//! a dedicated OS thread that owns the `MlxModelWeights` + `GpuContext` and
//! accepts requests over a `tokio::sync::mpsc` channel.
//!
//! # Why a channel + thread, not a `tokio::Mutex`
//!
//! Two equivalent designs were considered:
//!
//!   - **Mutex guarding (weights, ctx)** — every handler `.lock().await`s and
//!     runs the forward pass inside the critical section with
//!     `tokio::task::block_in_place`. This bleeds sync compute into the tokio
//!     task pool and requires a multi-thread runtime invariant.
//!   - **Worker thread + mpsc (this file)** — handlers send requests to a
//!     channel. The worker thread drains the channel serially and replies via
//!     `oneshot`. Compute is a plain `std::thread` so the tokio runtime is
//!     never blocked and the FIFO ordering is inherent.
//!
//! The second is chosen because (a) forward passes are ~10-100ms of pure
//! compute — holding a tokio mutex across that would starve keep-alive /
//! request-id / CORS layers; (b) the queue cap (Decision #19) maps directly
//! to the channel capacity; (c) it avoids the `block_in_place` footgun.
//!
//! # Reference lineage
//!
//! The prefill / decode / tokenize path is exactly the same pipeline as
//! `serve::cmd_generate` (see `/opt/hf2q/src/serve/mod.rs`). This module
//! does not reimplement the forward pass; it wraps it. Every existing
//! behavior (ADR-009 dense-KV, ADR-010 Q8 rerank, chat-template priority
//! order) is preserved by construction.

use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex};
use std::thread::JoinHandle;
use std::time::{Duration, Instant};

use anyhow::{Context, Result};
use tokenizers::Tokenizer;
use tokio::sync::{mpsc, oneshot};

use crate::inference::models::gemma4::{MlxModelWeights, ProfileAccumulator};
use crate::serve::config::Gemma4Config;
use crate::serve::forward_prefill::SoftTokenInjection;
use crate::serve::gpu::GpuContext;
use crate::serve::header;
use crate::serve::load_info::{
    self, ArchFamily, ChatTemplateSource, LoadInfo, LoadInfoBuilder, MoeShape, TokenizerSource,
};
use crate::serve::sampler_pure::{self, SamplingParams as SamplerParams};
// ADR-040 Phase C iter-2a (C2b) — Scheduler wiring per Shape A
// (dossier `docs/research/adr040-c2-wiring-dossier-2026-05-24.md` §2.1).
// `FifoSchedulerAdapter` is the only concrete scheduler instantiated under
// the SerialFifo arm; `SlotAware` is rejected at `spawn_with_mode`
// (iter-1.5 F1) so the worker only ever sees a concrete FIFO. The
// `Scheduler` trait import gives `stats()` its method-call surface (the
// trait carries the public `stats(&self) -> SchedulerStats` signature
// even though `advance_after_*` deliberately lives on the concrete type
// per dossier §2.9).
use crate::serve::multi_seq_kv::SlotId;
use crate::serve::scheduler::{
    AdmitError, AdmitRequest, FifoSchedulerAdapter, InflightBatchedScheduler, Scheduler,
    SchedulerPolicy, SchedulerStats, SchedulerStep, SlotHandle, StepError,
};
// ADR-040 iter-2-decode-C-stream-tool-call (§6.1.48) — `MultiSeqError`
// import is now ONLY used inside `#[cfg(test)]` modules (the slot-aware
// streaming fn's last typed-error `CapabilityUnsupported` constructor
// in the production binary was REPLACED with the real Wave 3 W-B3
// `ToolCallStreamEmitter` plumbing this iter).  Gate the import behind
// `#[cfg(test)]` to keep `cargo build --release` lint-clean while
// preserving the test-module `super::MultiSeqError::CapabilityUnsupported
// { capability: "..." }` constructors that pin the typed-deferral labels.
#[cfg(test)]
use crate::serve::multi_seq_kv::MultiSeqError;

// ---------------------------------------------------------------------------
// Public types
// ---------------------------------------------------------------------------

/// Sampling parameters passed to the engine worker. Full Tier 2/3/4
/// surface plumbed from the request.
///
/// **Honored at decode time** (iter-94, iter-95):
/// - `temperature`, `top_p`, `top_k`, `repetition_penalty` —
///   routed through `sampler_pure::sample_token` over the live
///   logits whenever any field requests non-greedy sampling
///   (see `sample_logits` gate in `generate_once`).  All-default
///   request → on-GPU greedy argmax fast path (no logits readback).
/// - `max_tokens`, `stop_strings` — decode-loop terminators.
/// - `logit_bias` — additive bias applied to live logits before
///   `sampler_pure` (Tier 4, OpenAI semantics).
/// - `grammar` + `token_bytes` — when present, mask invalid tokens
///   per-step before sampling (iter-95 grammar-constrained decode,
///   gated on `response_format=json_object`/`json_schema`).
///
/// **Plumbed but NOT yet honored** (accepted from the request,
/// retained on the struct, but not consumed by the current sampler):
/// - `frequency_penalty`, `presence_penalty` — Tier 2 OpenAI extras.
/// - `min_p` — Tier 3 llama.cpp extension.
/// - `seed` — RNG seeding (sampler_pure uses a thread-local RNG today).
/// - `logprobs`, `top_logprobs` — Tier 4 response shape; surface only.
/// Tool-call enforcement policy derived from the request's `tool_choice`.
///
/// Wave-2.5 A4: the streaming worker's `route_content` fallback silently
/// emitted unparseable tool-call bodies as plain Content for ALL tool_choice
/// modes, including `Required` and explicit `Function`.  For constrained
/// modes the model is supposed to emit a valid call (the grammar guarantees
/// it structurally), so a parse failure is a server-side bug — not a
/// graceful-degradation scenario.  `Auto` genuinely needs the fallback
/// because there is no grammar constraint and a partial/malformed tool call
/// is recoverable by the client as plain text.
///
/// Wave 3 W-B2 — `AutoLazyGrammar` variant added.  Once the W-B2 lazy
/// grammar wiring lands (`compile_tool_grammar` returns `Some(grammar)`
/// for `tool_choice=auto` with tools[] non-empty AND a registered
/// family), Auto becomes a constrained mode FROM `ToolCallOpen` ONWARDS.
/// The model is free to emit preamble content; once the open marker
/// fires the grammar enforces every body byte.  In that scenario a
/// body-parse-failure is structurally impossible — exactly the same
/// invariant that promotes Constrained's body-parse failure to a loud
/// error.  `AutoLazyGrammar` carries that contract through to the
/// streaming `emit_streaming_tool_call_close` and the non-streaming
/// `extract_tool_calls_from_text` so the loud-error promotion fires
/// equally on Required/Function AND Auto-with-grammar paths.
///
/// `Auto` (no grammar) keeps the content-fallback semantics — only fires
/// when the request is Auto AND tools[] is empty OR the model family is
/// unknown OR `tool_choice` was Auto and `compile_tool_grammar` returned
/// `Ok(None)`.  Under that branch there is no grammar enforcement, the
/// model may legitimately emit malformed syntax, and re-emitting the
/// raw bytes as Content is the right semantics.
///
/// # Why not derive from `schema::ToolChoiceValue` here
///
/// `SamplingParams` is an engine-layer type; it must not import from
/// `schema` (HTTP-layer).  This enum re-expresses only the
/// policy-relevant distinctions (Auto / AutoLazyGrammar / Constrained).
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub enum ToolCallPolicy {
    /// `tool_choice = "auto"` (or absent) AND no compiled grammar.  Grammar
    /// is optional under Auto, so this branch fires when:
    ///   * `tools[]` is empty / missing, OR
    ///   * the model family has no registered tool-call emitter, OR
    ///   * `tool_choice` was explicitly `"auto"` and the request never
    ///     declared tools.
    ///
    /// Body-parse failures fall back to Content (existing wave-2.5
    /// behaviour). Mirrors llama.cpp's unconstrained tool-call path.
    #[default]
    Auto,
    /// Wave 3 W-B2 — `tool_choice = "auto"` AND a lazy grammar IS active.
    /// `compile_tool_grammar` produced a `GrammarKind::ToolCallBodyAuto`
    /// runtime that is suspended (`awaiting_trigger=true`) until the
    /// `ToolCallSplitter` reports `ToolCallOpen`; from that point onwards
    /// the grammar enforces every body byte exactly like the Constrained
    /// path. A parse failure under this policy means the lazy grammar
    /// engine produced structurally invalid output — same regression
    /// signature as Constrained, same loud-error promotion required.
    ///
    /// Mirrors llama.cpp `grammar_lazy=true` at common/chat.cpp:898-913,
    /// 1177-1200, 1399-1416, 1626-1628.
    AutoLazyGrammar,
    /// `tool_choice = "required"` or `tool_choice = {type: "function", ...}`.
    /// Grammar guarantees well-formed output FROM BYTE 0; a parse failure is
    /// promoted to `GenerationEvent::Error` with `finish_reason = "error"`
    /// on the streaming path, and an HTTP 500 on the non-streaming path.
    Constrained,
}

impl ToolCallPolicy {
    /// `true` when the policy carries an active grammar that physically
    /// constrains the tool-call body (Constrained from byte 0, or
    /// AutoLazyGrammar from `ToolCallOpen` onwards).  Body-parse failures
    /// under these policies are unreachable in correct operation and
    /// promote to loud errors. Wave 3 W-B2 — single source of truth for
    /// the loud-error decision used by both streaming
    /// (`emit_streaming_tool_call_close`) and non-streaming
    /// (`extract_tool_calls_from_text`) paths.
    pub fn enforces_body_grammar(&self) -> bool {
        matches!(
            self,
            ToolCallPolicy::Constrained | ToolCallPolicy::AutoLazyGrammar
        )
    }
}

/// Kind discriminant for the grammar attached to a request.
///
/// Mirrors llama.cpp `enum common_grammar_type` at
/// `/opt/llama.cpp/common/common.h:171-176`:
///
/// ```c++
/// enum common_grammar_type {
///     COMMON_GRAMMAR_TYPE_NONE,
///     COMMON_GRAMMAR_TYPE_USER,
///     COMMON_GRAMMAR_TYPE_OUTPUT_FORMAT,
///     COMMON_GRAMMAR_TYPE_TOOL_CALLS,
/// };
/// ```
///
/// Wave 2.6 W-α5 motivation (cfa-20260427-adr005-wave2.6 research-report.md
/// Q1, audit `codex-review-last.txt` divergence "A1 / response_format
/// regression" severity HIGH): without this kind, the wave-2.5 A1 fix
/// gates **every** grammar on `ToolCallSplitter::in_tool_call()` — which
/// silently disables `response_format=json_object` / `json_schema`
/// enforcement on registered Gemma/Qwen models because the splitter never
/// fires for non-tool requests.  The kind tells the runtime whether to
/// enforce unconditionally (`ResponseFormat`) or to wait for a trigger
/// before enforcing (`ToolCallBody`, the lazy-grammar pattern from
/// llama.cpp PR #9639).
///
/// vLLM's `StructuredOutputsParams` and SGLang's mutually exclusive
/// `json_schema` / `regex` / `ebnf` fields are the same shape — one
/// constraint kind per request, asserted at request-parse time.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub enum GrammarKind {
    /// User-supplied or `response_format`-derived grammar.
    /// Applies UNCONDITIONALLY for the entire generation, from the very
    /// first token.  The grammar runtime never enters `awaiting_trigger`
    /// state — the mask fires every step and `accept_bytes` advances
    /// every step.  Pre-A1 (wave 2.4 and earlier) behavior.
    ///
    /// Mirrors `COMMON_GRAMMAR_TYPE_USER` + `COMMON_GRAMMAR_TYPE_OUTPUT_FORMAT`
    /// in llama.cpp.
    ///
    /// Default — preserves backward compatibility for any caller that
    /// constructs `SamplingParams` without setting the field.
    #[default]
    ResponseFormat,
    /// Tool-call body grammar under `tool_choice = auto`.
    ///
    /// Applies ONLY after the model emits the per-model open marker
    /// (e.g. Gemma 4 `<|tool_call>`, Qwen 3.5/3.6 `<tool_call>`).  Until
    /// then, the runtime sits in `awaiting_trigger == true`: `apply()`
    /// is a no-op (no mask), `accept()` is a no-op (no advance, no
    /// dead/accepted-state termination check).  When the
    /// ToolCallSplitter sees the open marker, the engine calls
    /// `runtime.trigger()` to flip the flag false; the runtime then
    /// enforces every subsequent token through to the close marker.
    ///
    /// Mirrors llama.cpp `grammar_lazy = true` for `tool_choice == AUTO`
    /// at `/opt/llama.cpp/common/chat.cpp:913, 1200, 1416`.
    ///
    /// Wired post-wave-3 W-B2 + Wave 3.5 HIGH-1: `compile_tool_grammar`'s
    /// Auto branch produces this kind whenever `tool_choice=Auto` AND
    /// `tools[]` is non-empty AND the model family is registered.  The
    /// grammar shape is `OneOrMoreCallsBodyOnly` (Wave 3.5 HIGH-1) — the
    /// first open marker is consumed by `ToolCallSplitter` before
    /// `runtime.trigger()` fires, so the grammar root expects body bytes
    /// (NOT the already-consumed marker).  Inter-call open markers
    /// (parallel) and the close marker remain part of the grammar.
    ///
    /// Auto with `tools[]` empty OR an unregistered model family stays
    /// at the prior `Ok(None)` no-grammar branch (open by design — Auto
    /// allows zero-call runs and there is no per-model wrapper to
    /// constrain).
    ToolCallBodyAuto,
    /// Tool-call body grammar under `tool_choice = required` or
    /// `tool_choice = function(name)`.
    ///
    /// EAGER from token 0 — the runtime never enters `awaiting_trigger`
    /// state; the mask fires every step starting at the very first
    /// decode token.  The grammar root is shape `OneOrMoreCalls` with the
    /// per-model open marker, body, and close marker all wired into the
    /// root rule (see `registry::GrammarShape`).  The model is
    /// structurally unable to emit non-call output: byte 0 must be the
    /// first byte of the open marker (e.g. `<` for Gemma 4 `<|tool_call>`)
    /// or the request rejects every other token via the mask.
    ///
    /// Mirrors llama.cpp `grammar_lazy = false` for
    /// `tool_choice == REQUIRED` at `/opt/llama.cpp/common/chat.cpp:898-913,
    /// 1177-1200, 1399-1416`.  Wave 2.7 W-η HIGH-1.
    ToolCallBodyRequired,
}

/// - `parallel_tool_calls` — Tier 4; lands with the tool-call path
///   referenced at the worker dispatch site (see Decision #21 in the
///   registration block).
#[derive(Debug, Clone)]
pub struct SamplingParams {
    pub temperature: f32,
    pub top_p: f32,
    pub top_k: usize,
    pub repetition_penalty: f32,
    pub max_tokens: usize,
    /// Stop strings — if one appears in the running decoded text, generation
    /// halts with finish_reason `stop`. Case-sensitive.
    pub stop_strings: Vec<String>,

    // --- Tier 2 additions (plumbed, not all wired into sampler yet) ---
    /// Nucleus-sampling lower bound used with `top_p`. OpenAI Tier 2.
    pub frequency_penalty: f32,
    /// OpenAI Tier 2.
    pub presence_penalty: f32,
    /// Optional RNG seed for reproducible sampling. `None` → thread RNG.
    /// Greedy (T=0) decodes are deterministic regardless.
    pub seed: Option<u64>,

    // --- Tier 3 addition (llama.cpp / ollama extension) ---
    /// Min-p sampling cutoff. `0.0` disables. Tier 3.
    pub min_p: f32,

    // --- Tier 4 (power-user) ---
    /// Per-token-id logit bias map. Additive bias applied to the
    /// live logits before `sampler_pure::sample_token` (wired in
    /// iter-94; OpenAI semantics — non-finite bias on a token vetoes
    /// it, finite bias shifts its logit).
    pub logit_bias: std::collections::HashMap<u32, f32>,
    /// If `true`, include top-k logprobs in the response. Tier 4.
    pub logprobs: bool,
    /// Number of top alternatives to report per chosen token. 0 = only the
    /// chosen token's logprob.
    pub top_logprobs: u32,
    /// `true` = allow multiple tool calls in the same turn. Tier 4. Plumbs
    /// through to the grammar-constrained decode path when it lands.
    pub parallel_tool_calls: bool,

    // --- Grammar-constrained decoding (Decision #6, Task #5, iter-95) ---
    /// Pre-compiled GBNF grammar to constrain decode-time token selection.
    /// `None` ⇒ unconstrained (default sampling on raw logits).  When
    /// `Some(g)`, the decode loop builds a fresh `GrammarRuntime`
    /// (`mask::mask_invalid_tokens` clones it per-token), calls the
    /// mask BEFORE `sampler_pure::sample_token`, and feeds the chosen
    /// token's bytes through the runtime so the next step's mask is
    /// correctly narrowed.
    ///
    /// Built by `handlers.rs::compile_response_format`.  `Grammar` is
    /// `Clone` (cheap — just a `Vec<Vec<GretElement>>`); the chat
    /// handler clones it into the per-request `SamplingParams`.
    pub grammar: Option<super::grammar::Grammar>,
    /// Per-vocab decoded UTF-8 byte table for grammar masking.  `None`
    /// when `grammar` is also `None` (no grammar, no need for the
    /// table).  When `grammar` is `Some`, this MUST be `Some(table)`
    /// — the chat handler obtains it via `Engine::token_bytes_table()`
    /// (lazily built + cached on the Engine).  Cheap to attach: an
    /// Arc clone (no copy of the underlying vector).
    pub token_bytes: Option<Arc<Vec<Vec<u8>>>>,
    /// Kind discriminant for the grammar attached above.
    ///
    /// Wave 2.6 W-α5 (research-report.md Q1, audit divergence "A1 /
    /// response_format regression").  Decides whether the grammar runtime
    /// is unconditionally enforcing (`ResponseFormat`, the default) or
    /// trigger-gated (`ToolCallBody`).  Set by:
    ///   * `compile_response_format` → `GrammarKind::ResponseFormat`
    ///   * `compile_tool_grammar`    → `GrammarKind::ToolCallBodyRequired`
    ///                                 (or `ToolCallBodyAuto` when wave 2.7+
    ///                                  wires AUTO to a marker-aware lazy
    ///                                  grammar — not yet reachable)
    ///
    /// Default = `ResponseFormat` so any caller that builds
    /// `SamplingParams` without touching the field gets the
    /// pre-wave-2.5 unconditional-enforcement behavior.
    pub grammar_kind: GrammarKind,

    // --- Wave-2.5 A4 — Tool-call parse-failure policy ---
    /// Policy for handling tool-call body parse failures.  Set from
    /// `tool_choice` by `handlers.rs::prepare_chat_generation_core`.
    /// Defaults to `Auto` (content fallback) so the pre-wave-2.5
    /// behavior is preserved for all callers that don't set this field.
    pub tool_call_policy: ToolCallPolicy,

    // --- ADR-005 iter-230 B — reasoning forced-open seed ---
    /// `true` when the rendered chat prompt ends inside an OPEN
    /// reasoning block (e.g. Qwen 3.6 template seeds `<think>\n` with
    /// thinking on), so the completion begins inside reasoning and the
    /// model never re-emits the open marker. Consumed by every
    /// `registry::make_reasoning_splitter` call on the generate paths.
    /// Computed by `handlers.rs` via
    /// `registry::prompt_seeds_reasoning_open` on the rendered prompt;
    /// `false` (the default) = pre-iter-230 behavior.
    pub reasoning_forced_open: bool,
}

impl Default for SamplingParams {
    /// Sampling defaults used when a request omits a field. T=0 greedy, no
    /// penalties. Matches the behavior of `cmd_generate` when all CLI
    /// sampling flags default.
    fn default() -> Self {
        Self {
            temperature: 0.0,
            top_p: 1.0,
            top_k: 0,
            repetition_penalty: 1.0,
            max_tokens: 512,
            stop_strings: Vec::new(),
            frequency_penalty: 0.0,
            presence_penalty: 0.0,
            seed: None,
            min_p: 0.0,
            logit_bias: std::collections::HashMap::new(),
            logprobs: false,
            top_logprobs: 0,
            parallel_tool_calls: true,
            grammar: None,
            token_bytes: None,
            grammar_kind: GrammarKind::default(),
            tool_call_policy: ToolCallPolicy::Auto,
            reasoning_forced_open: false,
        }
    }
}

/// Effective repetition penalty for sampling (2026-08-03 loop mitigation).
///
/// Client-supplied values (≠ 1.0) always win.  When the client omits
/// `repetition_penalty` (handler default `1.0`), fall back to the
/// server-wide `HF2Q_DEFAULT_REPETITION_PENALTY` (default `1.0` = off).
///
/// Shared by every arch's sampler-construction site (gemma engine.rs,
/// engine_qwen35.rs, engine_qwen3vl.rs) so the semantics are uniform
/// regardless of which model serves the request.
///
/// Applied ONLY at sampler-construction boundaries: `SamplingParams` is
/// never mutated, so every `repetition_penalty != 1.0` predicate
/// (`sample_logits` gates, cache bypasses, `is_greedy_eligible`) sees the
/// client's literal value and behaves exactly as before.  Pure-greedy
/// requests (T=0, all defaults) never reach a sampler on any arch — the
/// `sample_logits` predicates stay false and the GPU argmax path is
/// untouched.  Penalty scope downstream is the response's generated
/// tokens only, never the prompt — safe for code.
pub fn effective_repetition_penalty(params: &SamplingParams) -> f64 {
    if params.repetition_penalty != 1.0 {
        params.repetition_penalty as f64
    } else {
        crate::debug::INVESTIGATION_ENV.default_repetition_penalty as f64
    }
}

/// Owned soft-token override sent through the worker channel.
///
/// Identical contract to [`SoftTokenInjection`] but owns the
/// `MlxBuffer` (channel-friendly: needs `Send`).  The worker thread
/// rebuilds borrowed `SoftTokenInjection<'_>` slices from a
/// `&[SoftTokenData]` for the prefill call.  Phase 2c Task #17 / iter-98.
#[derive(Debug, Clone)]
pub struct SoftTokenData {
    /// Half-open position range within the prompt: `[start, end)`.
    pub range: std::ops::Range<usize>,
    /// Replacement embeddings, shape `[range.len(), hidden_size]` F32,
    /// row-major.  Cheap-clone (Arc-shared underlying Metal buffer).
    pub embeddings: mlx_native::MlxBuffer,
}

/// Owned, channel-friendly mirror of
/// [`crate::serve::forward_prefill::DeepstackInjection`] (ADR-005
/// iter-224 Wedge-4d).  Built at the chat-handler engine seam from
/// [`compute_vision_embeddings_gpu_qwen3vl`]'s augmented embed (one
/// chunk per Qwen3-VL DeepStack head). The worker thread rebuilds
/// borrowed `DeepstackInjection<'_>` slices from a `DeepstackData` for
/// the LM-side `forward_gpu_last_logits_with_soft_tokens_and_deepstack`
/// call.
#[derive(Debug, Clone)]
pub struct DeepstackData {
    /// Image-token positions in the post-`<|image_pad|>`-expansion
    /// prompt (concatenated across all images in the request, in
    /// natural left-to-right order).  Same length as the row count of
    /// every chunk.
    pub image_token_positions: Vec<u32>,
    /// One GPU buffer per ds layer, each shape `[n_image_tokens,
    /// hidden_size]` F32 row-major.  `chunks.len()` = n_deepstack;
    /// `chunks[i]` is added at LM layer `i`.
    pub chunks: Vec<mlx_native::MlxBuffer>,
}

impl DeepstackData {
    /// Number of deepstack layers (= chunks.len()).
    pub fn n_deepstack(&self) -> usize {
        self.chunks.len()
    }
    /// Number of image tokens (= image_token_positions.len()).
    pub fn n_image_tokens(&self) -> usize {
        self.image_token_positions.len()
    }
}

/// Result of a non-streaming chat generation.
#[derive(Debug, Clone)]
pub struct GenerationResult {
    /// Decoded text that goes into `message.content` — post reasoning-marker
    /// split (Decision #21). If the model has no reasoning markers
    /// registered, this is the full raw decoded text.
    pub text: String,
    /// Decoded text that goes into `message.reasoning_content`. `None` when
    /// the model's registration has no reasoning markers or when no
    /// reasoning span was emitted.
    pub reasoning_text: Option<String>,
    /// Prompt token count (after chat-template rendering + tokenization).
    pub prompt_tokens: usize,
    /// Completion token count (tokens emitted by the decoder).
    pub completion_tokens: usize,
    /// Number of completion tokens that were emitted inside a reasoning
    /// span (Decision #21). `None` when no reasoning markers registered /
    /// no reasoning span opened. Counted per-token in the decode loop.
    pub reasoning_tokens: Option<usize>,
    /// Reason generation halted: `"stop"` | `"length"`.
    pub finish_reason: &'static str,
    /// Prefill wall-clock.
    pub prefill_duration: Duration,
    /// Decode wall-clock.
    pub decode_duration: Duration,
    /// Number of prompt tokens served from the prompt cache (Phase 2a
    /// Task #7, Decision #24).  Reported via OpenAI's
    /// `usage.prompt_tokens_details.cached_tokens`.  Iter-96 single-slot
    /// full-equality cache: this is `prompt_tokens` on a cache hit
    /// (entire prefill + decode skipped) and `0` otherwise.  Iter-97+
    /// extends to LCP-based partial-prefill resume which can report
    /// any value `0 ≤ cached_tokens ≤ prompt_tokens`.
    pub cached_tokens: usize,
    /// ADR-020 AC#7 — per-completion-token log-probabilities under the
    /// model's RAW (pre-temperature/pre-rep-penalty) softmax.  `None`
    /// when the request did not set `logprobs: true`; otherwise length
    /// equals `completion_tokens`.  Populated in the decode loop via
    /// [`crate::serve::sampler_pure::sample_token_with_logprob`].
    /// Consumed by the response builder to populate
    /// [`crate::serve::api::schema::ChoiceLogprobs`].
    pub logprobs: Option<Vec<f32>>,
}

// ---------------------------------------------------------------------------
// Engine handle (the public API)
// ---------------------------------------------------------------------------

/// Engine handle — cheap to clone, threaded through `AppState`. All methods
/// are async so they can be awaited from axum handlers without blocking the
/// tokio runtime.
#[derive(Clone)]
pub struct Engine {
    inner: Arc<EngineInner>,
}

/// Iter-215 Wedge-2: surface for which `LoadedModel` variant the
/// engine wraps.  Handlers that bypass the worker round-trip (e.g.
/// stream-mode chat, where the worker emits Error events into a
/// separate channel rather than returning a single Result) inspect
/// this to dispatch the HTTP 501 short-circuit at the handler layer.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum LoadedArch {
    /// Gemma 4 (and Gemma-shaped) GGUFs.  Production chat path.
    Gemma,
    /// Qwen3.5 / Qwen3.6 (dense + MoE).  Iter-215 MVP returns 501 on
    /// chat / embed / vision; Wedge-3 wires the live forward pass.
    Qwen35,
    /// Qwen3-VL text-LM (ADR-005 Wedge-4 / iter-228a). MVP returns 501 on
    /// chat / embed / vision via the
    /// [`crate::inference::models::qwen3vl_text::forward::QWEN3VL_TEXT_FORWARD_PENDING_SENTINEL`]
    /// sentinel; iter-228b wires the live dense transformer forward.
    Qwen3VlText,
    /// DeepSeek-V4-Flash native compressed-attention runtime.
    Deepseek4,
}

// ---------------------------------------------------------------------------
// ADR-040 Phase C iter-1 — EngineMode scaffolding (2026-05-23)
//
// This block introduces the EngineMode enum + signature-only constructor
// extension. Per ADR-040 §2.1 + §3.6 + AC-3, iter-1 ships scaffolding only:
//   - `SerialFifo` (default) preserves ADR-005 Decision #2 + #19 behaviour
//     byte-for-byte (one mpsc channel, one worker thread, serialized FIFO
//     dispatch, 429 + Retry-After on queue overflow).
//   - `SlotAware { max_slots }` is the new ADR-040 path. At iter-1 it is
//     SIGNATURE-ONLY — `spawn_with_mode` delegates to the existing
//     `spawn` regardless of mode. Phase C iter-2 forks the SlotAware path
//     to use `crate::serve::scheduler::Scheduler` once Phase A iter-2+
//     (`MultiSeqKvCache` per-model impls) lands.
//
// The byte-equivalence regression pin lives in
// `adr040_phase_c_iter1_engine_mode_tests::engine_spawn_signature_unchanged_at_phase_c_iter_1`
// — a compile-time gate that fails if the production 3-arg `Engine::spawn`
// signature is ever modified by a future iter.
// ---------------------------------------------------------------------------

/// ADR-040 Phase C iter-1: which scheduling model the engine uses.
///
/// `SerialFifo` (default) preserves ADR-005 Decision #2 + #19 behaviour
/// byte-for-byte: one mpsc channel, one worker thread, serialized FIFO
/// dispatch, 429 + Retry-After on queue overflow.
///
/// `SlotAware { max_slots }` is the new ADR-040 path — admits up to
/// `max_slots` concurrent requests against a multi-seq KV cache. This
/// variant is SIGNATURE-ONLY at Phase C iter-1; production routing
/// activates in iter-2 once `crate::serve::scheduler::Scheduler` is
/// wired through `Engine::spawn` and the per-model `MultiSeqKvCache`
/// impls land (Phase A iter-2+).
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum EngineMode {
    /// ADR-005 Decision #2 + #19 production path. One mpsc channel, one
    /// worker thread, serialized FIFO dispatch. Iter-1 default.
    SerialFifo,
    /// ADR-040 Phase C iter-2+ slot-aware path. Admits up to `max_slots`
    /// concurrent requests against a multi-seq KV cache.
    SlotAware {
        /// Concurrency bound, per ADR-040 §3.4 default `max_slots = 4`.
        max_slots: u32,
    },
}

impl Default for EngineMode {
    /// ADR-040 §3.6 — `SerialFifo` is the production default. This preserves
    /// the ADR-005 Phase 2 contract byte-for-byte until Phase E1's measured
    /// cutover gate fires.
    fn default() -> Self {
        Self::SerialFifo
    }
}

/// ADR-040 Phase C iter-1.5 — typed error returned by
/// [`Engine::spawn_with_mode`] when the requested mode is not yet wired.
///
/// At iter-1 the `SlotAware` runtime is signature-only; rather than silently
/// degrading to `SerialFifo` (a Liskov-substitution violation per the
/// adversarial review of iter-1), the constructor now FAILS FAST with this
/// error. Iter-2 lands the `Scheduler` + `MultiSeqKvCache` wiring and removes
/// the rejection from `spawn_with_mode`.
///
/// Per ADR-040 §7 "no fallback, no stub (todo later) code": the prior
/// `let _ = mode;` discard was a stub; this typed error is the honest
/// surface until iter-2 ships.
///
/// # Iter-status (post-C2b)
///
/// As of C2b (commit `886f229c`, 2026-05-23) the `worker_run` refactor has
/// shipped: the worker thread now constructs a `FifoSchedulerAdapter` at
/// entry and wraps every dispatch arm in admit→drive→release. SerialFifo
/// remains the only mode that survives spawn-time validation — SlotAware
/// is still rejected here because:
///   - **iter-2b (Qwen35 worker arm)** wires the slot-aware decode loop
///     for the Qwen3.5/3.6 family on top of B4b's decode-side `slot_id`
///     threading.
///   - **iter-2c (Gemma 4 worker arm)** does the same for the Gemma 4
///     family on top of B4c's `forward_prefill.rs` + `forward_prefill_
///     batched.rs` slot-id threading.
/// Until both per-family arms land, `--scheduler inflight_batched` /
/// `HF2Q_SCHEDULER=inflight_batched` will surface this error via
/// `load_engine`'s `anyhow::Error` wrapper and `cmd_serve` will exit
/// non-zero before binding the listener (fail-loud per §7 mantra; no
/// silent fallback to SerialFifo).
#[derive(Debug, thiserror::Error)]
pub enum EngineSpawnError {
    /// The requested [`EngineMode`] variant requires runtime support that
    /// has not yet landed. Carries the iter that introduced the surface
    /// (`iter_landed`, e.g. `"C2b"` for the worker-refactor commit) and
    /// the iter that will implement the per-family runtime
    /// (`iter_required`, e.g. `"C2b/C2c (per-family)"`) so callers +
    /// tooling get a precise diagnostic. The `Display` impl names the
    /// commit + the two per-family follow-up iters so operator log greps
    /// can match on either the phase letter or the per-family iter name.
    #[error(
        "ADR-040 EngineMode::SlotAware not yet wired (Phase {iter_landed} \
         landed; Phase {iter_required} implements). C2b SHIPPED at 886f229c \
         (worker_run + FifoSchedulerAdapter wiring); SlotAware runtime gated \
         on iter-2b (Qwen35 worker arm, B4b decode-slot threading) + iter-2c \
         (Gemma 4 worker arm, B4c prefill-slot threading). \
         Use `--scheduler fifo_serial` (or unset `HF2Q_SCHEDULER`), or wait \
         for Phase {iter_required}."
    )]
    ModeNotYetWired {
        /// The iter that introduced the API surface (e.g. `"C2b"`).
        iter_landed: &'static str,
        /// The iter that will implement the runtime (e.g.
        /// `"C2b/C2c (per-family worker arms)"`).
        iter_required: &'static str,
    },

    /// **ADR-040 Phase C iter-2c (C2c)** — `EngineMode::SlotAware`
    /// spawn for Gemma 4 reached the multi-seq KV provisioning step
    /// (per-layer `MultiSeqHbKvBuffers` via the A3a allocator with
    /// `n_seqs = max_slots`) but the allocator returned an error.
    /// Distinct from [`Self::ModeNotYetWired`] (mode not implemented)
    /// — this surface is reached only when the implementation is wired
    /// AND a real per-layer allocation failed (typical causes: MlxDevice
    /// OOM at production shape × N slots; per-layer `nkv/hd/cap` zero
    /// because a malformed `Gemma4Config`).
    #[error(
        "ADR-040 C2c: Gemma 4 SlotAware spawn failed during multi-seq KV \
         provisioning (max_slots={max_slots}). Cause: {cause}"
    )]
    Gemma4SlotAwareProvisionFailed {
        /// The `max_slots` value the caller requested.
        max_slots: u32,
        /// First per-layer allocator failure as rendered by anyhow.
        cause: String,
    },

    /// **ADR-040 Phase C iter-2d (C2d)** — `EngineMode::SlotAware`
    /// spawn for Qwen35 reached the multi-seq KV provisioning step
    /// (single `HybridKvCache::new(.., n_seqs = max_slots)` covering all
    /// full-attn + linear-attn + optional MTP slots, per the A2a
    /// multi-seq lift) but the allocator returned an error. Distinct
    /// from [`Self::ModeNotYetWired`] (mode not implemented) — this
    /// surface is reached only when the implementation is wired AND a
    /// real per-layer allocation failed (typical causes: MlxDevice OOM
    /// at production shape × N slots; malformed `Qwen35Config` with
    /// zero `hidden_size` / `head_dim` / layer count).
    ///
    /// Mirrors `Gemma4SlotAwareProvisionFailed` for the Qwen35 family.
    /// Distinct discriminants so per-family handlers + log greps stay
    /// unambiguous.
    #[error(
        "ADR-040 C2d: Qwen35 SlotAware spawn failed during multi-seq KV \
         provisioning (max_slots={max_slots}). Cause: {cause}"
    )]
    Qwen35SlotAwareProvisionFailed {
        /// The `max_slots` value the caller requested.
        max_slots: u32,
        /// `HybridKvCache::new` failure as rendered by anyhow.
        cause: String,
    },

    /// **ADR-040 Phase C iter-C2c-cont** — `EngineMode::SlotAware`
    /// spawn for Gemma 4 reached the SIBLING `MultiSeqHybridKvBuffers`
    /// scaffold provisioning step (the PRODUCTION-DEFAULT KV path per
    /// H10 falsification at §6.1.11 — `HF2Q_HYBRID_KV` is default-true
    /// since ADR-029 iter-13, 2026-05-11) but the per-layer
    /// `alloc_multi_seq_hybrid_kv_for_layer` allocator returned an error.
    /// Distinct from [`Self::Gemma4SlotAwareProvisionFailed`] (which
    /// names the HbKvBuffers HB-encoded opt-out scaffold) so operator
    /// log greps + per-handler routing stay unambiguous about WHICH KV
    /// regime's allocator failed.
    ///
    /// Reached only when `INVESTIGATION_ENV.hybrid_kv == true` (which is
    /// the default since 2026-05-11 — the falsified opt-in assumption).
    /// Typical causes: MlxDevice OOM at production shape × N slots
    /// (F16-K + TQ-HB-V doubles the per-slot byte footprint vs HB-only);
    /// `HF2Q_FULL_F16_KV=1` further inflates V to full F16; xlen mode
    /// (`HF2Q_DFLASH_XLEN_SDPA=1`) adds two more BF16 buffers per slot.
    ///
    /// Mirrors `Gemma4SlotAwareProvisionFailed` shape; ships alongside
    /// it so the spawn-time provisioning surface emits ONE typed error
    /// per failing KV regime + the operator can grep on the variant
    /// discriminant.
    #[error(
        "ADR-040 iter-C2c-cont: Gemma 4 SlotAware spawn failed during \
         MultiSeqHybridKvBuffers (production-default per HF2Q_HYBRID_KV=1; \
         H10 falsification §6.1.11) provisioning (max_slots={max_slots}). \
         Cause: {cause}"
    )]
    Gemma4HybridSlotAwareProvisionFailed {
        /// The `max_slots` value the caller requested.
        max_slots: u32,
        /// First per-layer `alloc_multi_seq_hybrid_kv_for_layer`
        /// failure as rendered by anyhow.
        cause: String,
    },

    /// **ADR-040 Phase C iter-C2e (2026-05-30)** —
    /// `EngineMode::SlotAware` spawn for Qwen3-VL reached the
    /// witness provisioning step
    /// ([`super::engine_qwen3vl::Qwen3VlTextLoadedModel::provision_multi_seq_kv_for_slot_aware`])
    /// but the witness setter returned an error (only reachable today
    /// when the caller passed `max_slots == 0` past the spawn-arm's
    /// own pre-check — the spawn arm catches this first per the H218
    /// + H220 invariants and surfaces
    /// [`Self::ModeNotYetWired`] instead).
    ///
    /// Mirrors [`Self::Gemma4SlotAwareProvisionFailed`] +
    /// [`Self::Qwen35SlotAwareProvisionFailed`] shape and gives
    /// per-family discriminants for log greps + operator triage.
    ///
    /// **Forward-pointer (iter-C2e-cont, post iter-228a)**: once the
    /// iter-228a 501 sentinel is replaced with a real Qwen3-VL
    /// forward path landing a persistent KV cache, this variant will
    /// become reachable via real `MlxDevice` OOM at production
    /// shape × N slots (mirror of the Qwen35 OOM surface). For now
    /// the variant exists so the typed surface stays symmetric with
    /// Gemma 4 + Qwen35 — the C2e arm of `spawn_with_mode` calls the
    /// witness provisioner under the same `if let Err(e) = ...`
    /// pattern its siblings use.
    #[error(
        "ADR-040 C2e: Qwen3-VL SlotAware spawn failed during multi-seq KV \
         provisioning (max_slots={max_slots}). Cause: {cause}"
    )]
    Qwen3VLSlotAwareProvisionFailed {
        /// The `max_slots` value the caller requested.
        max_slots: u32,
        /// Witness provisioner failure (or post iter-C2e-cont: first
        /// per-layer allocator failure) as rendered by anyhow.
        cause: String,
    },

    /// **ADR-040 Phase A4 iter-1 (2026-05-30)** — `EngineMode::SlotAware
    /// { max_slots: N }` exceeded the published spec-decode safe-zone
    /// inflection point.
    ///
    /// Per the §6.1.53 + §6.1.54 deep-research dossier
    /// ([`docs/research/adr040-a4-drafter-multi-seq-dossier-2026-05-30.md`]),
    /// 3 independent published sources confirm that speculative
    /// decoding **net-regresses above 4-8 concurrent requests**:
    ///
    /// | Concurrent batch | Spec-decode net effect |
    /// |---|---|
    /// | 1 | +2.5× (memory-bandwidth-bound) |
    /// | 2-4 | Net positive |
    /// | 4-8 | Transition zone |
    /// | 8-16 | Net regression (verification overhead consumes gains) |
    /// | 16-32+ | Compute-bound; spec-decode is dead weight |
    ///
    /// This typed error is the operator-facing guardrail: spawn fails
    /// LOUDLY (not silently degrades) when an operator selects
    /// `max_slots > HF2Q_MAX_BATCHED_SLOTS` (continuous-batching ceiling,
    /// default 8; legacy `HF2Q_SPEC_DECODE_MAX_BATCHED_SLOTS` still honoured)
    /// without explicitly opting in via
    /// `HF2Q_SPEC_DECODE_ALLOW_OVERSIZED=1`.
    ///
    /// **Why fail-loud**: per ADR-040 §7 "no fallback, no stub" mantra,
    /// silently capping `max_slots` to 4 would be a Liskov-substitution
    /// violation (caller asked for N, got 4 without notice).  The
    /// operator-facing fix is documented in the dossier §7 operator
    /// runbook: either lower `max_slots` to ≤4 OR opt into the
    /// documented regression by setting `HF2Q_SPEC_DECODE_ALLOW_OVERSIZED=1`.
    ///
    /// **Reopen conditions** (dossier §7): empirical hf2q
    /// inflection-point measurement on its own hardware, customer ask
    /// with documented safe-zone workload, EAGLE-4 or successor lands
    /// with a published higher-batch contract, hf2q switches primary
    /// model away from MoE, or hf2q ships a KV-quantization scheme
    /// that reduces verification overhead.
    ///
    /// **Distinct from**
    /// [`Self::ModeNotYetWired`]: SlotAware IS wired (C2c §6.1.21 +
    /// C2d §6.1.22 + C2e §6.1.52 shipped); this is the OVERSIZED
    /// guardrail.  Distinct from `Gemma4SlotAwareProvisionFailed` /
    /// `Qwen35SlotAwareProvisionFailed` / `Qwen3VLSlotAwareProvisionFailed`
    /// (real allocator failures); this is a pre-flight policy gate
    /// that fires BEFORE any per-arch provisioning runs.
    #[error(
        "ADR-040 §6.1.54 A4 iter-1: EngineMode::SlotAware {{ max_slots: {max_slots} }} \
         exceeds the published spec-decode safe-zone threshold (max_slots > {threshold}). \
         3 independent published sources confirm speculative decoding net-regresses \
         above 4-8 concurrent requests (cite: {cite}). \
         Fix: lower max_slots to ≤{threshold}, OR opt in explicitly via \
         HF2Q_SPEC_DECODE_ALLOW_OVERSIZED=1 (documented regression). \
         Per ADR-040 §7 no-fallback mantra: silent capping would be a Liskov violation."
    )]
    SpecDecodeMaxSlotsAboveBatchedThreshold {
        /// The `max_slots` value the caller requested.
        max_slots: u32,
        /// The threshold the request exceeded (default 4 per dossier
        /// §1.5 + §3 — the published spec-decode net-positive ceiling).
        /// Tunable via `HF2Q_SPEC_DECODE_MAX_BATCHED_SLOTS` for
        /// operators who have empirically measured a different
        /// workload-specific inflection point.
        threshold: u32,
        /// Citation to the dossier so operator log greps + triage
        /// paths land directly on the load-bearing research. Static
        /// string because the call site is always known at compile
        /// time — no allocation in the error path (same discipline as
        /// `MultiSeqError::CapabilityUnsupported`).
        cite: &'static str,
    },
}

// ──────────────────────────────────────────────────────────────────────────
// ADR-040 Phase A4 iter-1 (2026-05-30) — spec-decode max-slots policy env
// readers.  Pure functions for deterministic testing.
//
// The readers are intentionally per-call (not LazyLock) so an operator
// toggle takes effect at the next spawn — matches the existing
// `HF2Q_FULL_F16_KV` / `HF2Q_DFLASH_XLEN_SDPA` per-call read discipline
// in `alloc_multi_seq_hybrid_kv_for_layer` at
// `src/inference/models/gemma4/kv_cache.rs:1064-1068`.
// ──────────────────────────────────────────────────────────────────────────

/// ADR-040 Phase A4 iter-1 (2026-05-30) — default for
/// `HF2Q_SPEC_DECODE_MAX_BATCHED_SLOTS` per the §6.1.53 + §6.1.54
/// dossier §1.5 + §3 finding (spec-decode net-positive ceiling is 4-8
/// concurrent; conservative default is the safe-zone *lower* edge).
///
/// **FAIL-CLOSED, deliberately stays 4 (do NOT relax to 8).** This is the
/// gate for the SPEC-DECODE drafter path — when that drafter ships, its
/// verification overhead net-regresses above 4 concurrent (dossier). The
/// continuous-batching capacity ceiling is the SEPARATE
/// [`ADR040_F_DEFAULT_CONTINUOUS_BATCHING_MAX_SLOTS`] (= 8); the two were
/// decoupled 2026-06-24 (Phase F, codex milestone review of `b671dfe0`
/// SHIP-WITH-FIXES item (c)) so a future drafter implementer cannot
/// inherit the relaxed continuous-batching default for the actual
/// spec-decode path. `adr040_phase_f_gate_decoupling_pin` enforces the
/// separation at compile/test time.
pub const ADR040_A4_DEFAULT_SPEC_DECODE_MAX_BATCHED_SLOTS: u32 = 4;

/// ADR-040 Phase F (2026-06-24) — default for `HF2Q_MAX_BATCHED_SLOTS`,
/// the capacity ceiling for the WIRED continuous/inflight-batching
/// `SlotAware` path (operator request "support up to 8 concurrent slots ×
/// 32k context each").
///
/// 8 is the UPPER edge of the dossier's 4-8 safe zone and is empirically
/// validated for continuous batching (NOT spec-decode): byte-identical to
/// serial (`slot_aware_n1`/`slot_aware_n4` re-run through the batched body
/// with `HF2Q_BATCHED_BODY=1`, 2026-06-24 — see ADR §0.13 queen-led audit),
/// coherence-proven e2e, 202 tok/s aggregate.
///
/// KV-memory accounting (ADR-040 `iter-F-kvcap`, 2026-06-24): the multi-seq
/// `SlotAware` scaffold splits the full-attention context budget across slots
/// — each of `max_slots` slots gets `max_position_embeddings / max_slots` of
/// global-layer context (the llama.cpp `-c`÷`-np` convention) via
/// `layer_type_to_alloc_params_per_slot` (`kv_cache.rs`). The 25 sliding
/// layers stay at the 1024 ring window (per-slot-independent). So the total
/// full-attention KV is ≈ ONE full-context sequence regardless of N (constant,
/// not linear): at N=8 each slot holds 262144/8 = 32k of global context →
/// global layers ~4 GB total (8 × 5 × nkv=2 × hd=512 × cap=32768 ×
/// [2 B F16-K + 1 B TQ-HB-V]); the whole multi-seq KV is ~5-6 GB, and "8×32k"
/// is now literal. Total at N=8 ≈ 16.4 GB weights + ~6 GB KV ≈ 22 GB → fits
/// the 128 GB M5 Max with vast headroom, and now also fits a 64 GB machine.
/// (Pre-`iter-F-kvcap` each slot eagerly held the full 262k → ~45 GB at N=8,
/// the ~10×-too-large figure Worker C caught.) `max_slots=1` is identity
/// (max/1 = max) so SerialFifo / single-seq is unchanged. Over-budget spawns
/// still fail CLOSED with an `alloc_*_kv_for_layer` `Result` error (not UB).
/// This is the gate the live `SlotAware` spawn checks; the
/// spec-decode drafter (unwired, API-scaffold only per §6.1.55-F5) must
/// gate on [`ADR040_A4_DEFAULT_SPEC_DECODE_MAX_BATCHED_SLOTS`] (= 4) when
/// it lands — NOT this constant.
pub const ADR040_F_DEFAULT_CONTINUOUS_BATCHING_MAX_SLOTS: u32 = 8;

/// ADR-040 Phase A4 iter-1 (2026-05-30) — load-bearing dossier citation
/// for the [`EngineSpawnError::SpecDecodeMaxSlotsAboveBatchedThreshold`]
/// error variant.  Operator log greps land directly on the path so
/// triage routes to the research source not a code line.
pub const ADR040_A4_DOSSIER_CITE: &str = "ADR-040 §6.1.53 + §6.1.54 A4 dossier — \
     docs/research/adr040-a4-drafter-multi-seq-dossier-2026-05-30.md";

/// ADR-040 Phase A4 iter-1 (2026-05-30) — read
/// `HF2Q_SPEC_DECODE_MAX_BATCHED_SLOTS` returning the parsed threshold
/// or the default ([`ADR040_A4_DEFAULT_SPEC_DECODE_MAX_BATCHED_SLOTS`]).
///
/// Malformed env (non-numeric, zero, or overflow) falls back to the
/// default with a `tracing::warn!`.  Pure function — takes a closure
/// that yields the env value so tests can drive deterministic input
/// without touching process env.
pub fn read_spec_decode_max_batched_slots<F>(env_read: F) -> u32
where
    F: FnOnce(&str) -> Option<String>,
{
    match env_read("HF2Q_SPEC_DECODE_MAX_BATCHED_SLOTS") {
        Some(s) => match s.trim().parse::<u32>() {
            Ok(0) => {
                tracing::warn!(
                    target: "adr040.a4",
                    "HF2Q_SPEC_DECODE_MAX_BATCHED_SLOTS={s:?} parsed to 0 — \
                     ignoring (would block all SlotAware spawns); using \
                     default {default}",
                    default = ADR040_A4_DEFAULT_SPEC_DECODE_MAX_BATCHED_SLOTS
                );
                ADR040_A4_DEFAULT_SPEC_DECODE_MAX_BATCHED_SLOTS
            }
            Ok(n) => n,
            Err(_) => {
                tracing::warn!(
                    target: "adr040.a4",
                    "HF2Q_SPEC_DECODE_MAX_BATCHED_SLOTS={s:?} unparseable as u32; \
                     using default {default}",
                    default = ADR040_A4_DEFAULT_SPEC_DECODE_MAX_BATCHED_SLOTS
                );
                ADR040_A4_DEFAULT_SPEC_DECODE_MAX_BATCHED_SLOTS
            }
        },
        None => ADR040_A4_DEFAULT_SPEC_DECODE_MAX_BATCHED_SLOTS,
    }
}

/// ADR-040 Phase F (2026-06-24) — read the continuous/inflight-batching
/// `SlotAware` capacity ceiling: prefer `HF2Q_MAX_BATCHED_SLOTS`, default
/// [`ADR040_F_DEFAULT_CONTINUOUS_BATCHING_MAX_SLOTS`] (= 8).
///
/// This is the gate the live `SlotAware` spawn checks — DISTINCT from the
/// spec-decode drafter gate (codex `b671dfe0` review item (c): keep the
/// two fail-closed-separate so the future drafter can't inherit 8).  For
/// operator back-compat, when `HF2Q_MAX_BATCHED_SLOTS` is unset but the
/// legacy `HF2Q_SPEC_DECODE_MAX_BATCHED_SLOTS` is set, the legacy value is
/// honoured with a deprecation `warn!` (it used to drive this gate before
/// the decoupling).  Malformed/zero env falls back to the default.  Pure
/// function — closure-injected env for deterministic tests.
pub fn read_continuous_batching_max_slots<F>(env_read: F) -> u32
where
    F: Fn(&str) -> Option<String>,
{
    let parse = |s: String, var: &str| -> u32 {
        match s.trim().parse::<u32>() {
            Ok(0) => {
                tracing::warn!(
                    target: "adr040.f",
                    "{var}={s:?} parsed to 0 — ignoring (would block all \
                     SlotAware spawns); using default {default}",
                    default = ADR040_F_DEFAULT_CONTINUOUS_BATCHING_MAX_SLOTS
                );
                ADR040_F_DEFAULT_CONTINUOUS_BATCHING_MAX_SLOTS
            }
            Ok(n) => n,
            Err(_) => {
                tracing::warn!(
                    target: "adr040.f",
                    "{var}={s:?} unparseable as u32; using default {default}",
                    default = ADR040_F_DEFAULT_CONTINUOUS_BATCHING_MAX_SLOTS
                );
                ADR040_F_DEFAULT_CONTINUOUS_BATCHING_MAX_SLOTS
            }
        }
    };
    if let Some(s) = env_read("HF2Q_MAX_BATCHED_SLOTS") {
        return parse(s, "HF2Q_MAX_BATCHED_SLOTS");
    }
    if let Some(s) = env_read("HF2Q_SPEC_DECODE_MAX_BATCHED_SLOTS") {
        tracing::warn!(
            target: "adr040.f",
            "HF2Q_SPEC_DECODE_MAX_BATCHED_SLOTS is DEPRECATED for the \
             continuous-batching gate — use HF2Q_MAX_BATCHED_SLOTS. \
             Honouring legacy value for back-compat."
        );
        return parse(s, "HF2Q_SPEC_DECODE_MAX_BATCHED_SLOTS (deprecated)");
    }
    ADR040_F_DEFAULT_CONTINUOUS_BATCHING_MAX_SLOTS
}

/// ADR-040 Phase A4 iter-1 (2026-05-30) — read
/// `HF2Q_SPEC_DECODE_ALLOW_OVERSIZED` returning `true` when set to
/// `"1"` / `"true"` / `"on"` (matches the
/// `HF2Q_FULL_F16_KV` convention at `gemma4/kv_cache.rs:1066`).
///
/// Pure function — takes a closure that yields the env value so tests
/// can drive deterministic input without touching process env.
pub fn read_spec_decode_allow_oversized<F>(env_read: F) -> bool
where
    F: FnOnce(&str) -> Option<String>,
{
    env_read("HF2Q_SPEC_DECODE_ALLOW_OVERSIZED")
        .map(|v| matches!(v.trim(), "1" | "true" | "on"))
        .unwrap_or(false)
}

/// **ADR-040 §3.5 iter-A5b** — pre-stream admit-time errors surfaced by
/// [`Engine::try_admit_budget`].
///
/// The variant exists so the SSE handler can route per-slot KV-budget
/// rejections to `ApiError::slot_budget_exceeded` (HTTP 429 +
/// `Retry-After: 1`) BEFORE `generate_stream_with_deepstack` returns
/// `Ok` and the handler commits to opening an SSE body — the codex
/// review CRITICAL #2 finding (handlers.rs:1739-1748 only matched on
/// the `queue_full` anyhow prefix; SlotBudgetExceeded for the
/// streaming arm reached the operator as a half-rendered SSE error
/// frame instead of a clean 429).
///
/// The non-streaming admit path inside `worker_run` continues to surface
/// the typed [`crate::serve::scheduler::AdmitError::SlotBudgetExceeded`]
/// inside the existing anyhow channel-error envelope; the worker error
/// string carries the `"slot_budget_exceeded"` literal so the
/// non-streaming handler arm can string-match parallel to
/// `"queue_full"`. Pre-stream surfacing via this typed enum is the
/// streaming-arm fix; defense-in-depth at the worker layer is preserved.
#[derive(Debug, thiserror::Error)]
pub enum EngineAdmitError {
    /// The request's projected KV byte cost
    /// (`(prompt_tokens + max_tokens) × kv_bytes_per_token`) exceeds the
    /// per-slot KV budget configured at engine spawn time
    /// (`kv_cache_budget_bytes / max_slots`). Maps to HTTP 429 +
    /// `Retry-After: 1` upstream via
    /// [`crate::serve::api::schema::ApiError::slot_budget_exceeded`].
    ///
    /// Distinct from
    /// [`crate::serve::scheduler::AdmitError::QueueFull`] (transient
    /// — capacity will free) because this is operator-actionable on
    /// the REQUEST: reducing `max_tokens` or shortening the prompt is
    /// the fix.
    #[error(
        "ADR-040 §3.5 A5: slot_budget_exceeded — needed_bytes={needed_bytes}, \
         budget_bytes={budget_bytes}. Reduce max_tokens or use a shorter \
         prompt; per-slot budget = kv_cache_budget_bytes / max_slots."
    )]
    SlotBudgetExceeded {
        needed_bytes: u64,
        budget_bytes: u64,
    },
}

/// Iter-215 Wedge-2 test fixture — build a synthetic `Engine` with a
/// no-op worker thread reporting the requested `LoadedArch`.  Used by
/// router / handler tests in sibling modules to exercise the 501
/// short-circuit path without a live model + GPU + GGUF on disk.
///
/// The worker drains the channel and exits cleanly on `Shutdown`; it
/// drops every other request kind silently (the test code that builds
/// these engines either never sends a request, or doesn't await the
/// reply).
/// Phase B-dense.2 follow-up — test fixture exposed to sibling
/// crates/modules so the kv_persist::families::gemma4_dense tests can
/// build an `Engine` carrying a populated `KvSpillDescriptor` AND a
/// no-op KV worker that handles `KvSnapshot` / `KvRestore` requests
/// against an in-memory byte map.
///
/// The synthetic engine reports `LoadedArch::Gemma`, hands out the
/// supplied descriptor, and the worker handles only KV requests +
/// Shutdown — every other request kind is dropped silently (the
/// caller never awaits a reply for those).
///
/// `seeded_layers` populates the in-memory cache with a deterministic
/// byte pattern keyed on `(layer, head, slot, byte_index)` so the
/// caller can assert byte-exactness on round-trips. A layer not in
/// `seeded_layers` returns zero bytes from snapshot.
#[cfg(test)]
pub(crate) fn make_synthetic_kv_engine_for_test(
    descriptor: super::kv_spill_descriptor::KvSpillDescriptor,
    seeded_layers: Vec<(usize, u8)>,
) -> Engine {
    use std::collections::HashMap;

    let nkv: Vec<usize> = descriptor.nkv_heads.clone();
    let head_dim: Vec<usize> = descriptor.head_dim.clone();
    let capacity: Vec<usize> = (0..descriptor.num_layers)
        .map(|i| match descriptor.layer_types[i] {
            crate::serve::config::LayerType::Sliding => descriptor.sliding_window,
            crate::serve::config::LayerType::Full => descriptor.max_decode_tokens.max(64),
        })
        .collect();
    let is_sliding: Vec<bool> = descriptor
        .layer_types
        .iter()
        .map(|lt| *lt == crate::serve::config::LayerType::Sliding)
        .collect();
    let descriptor_for_engine = descriptor.clone();
    let kv_dtype_bytes = descriptor.kv_dtype.elem_bytes();

    let (tx, mut rx) = mpsc::channel::<Request>(8);
    let nkv_clone = nkv.clone();
    let head_dim_clone = head_dim.clone();
    let capacity_clone = capacity.clone();
    let is_sliding_clone = is_sliding.clone();
    let handle = std::thread::Builder::new()
        .name("hf2q-engine-kv-bridge-test".into())
        .spawn(move || {
            // (layer, head, slot) -> (k_bytes, v_bytes) — per-token
            // chunk size is `head_dim * elem_bytes`.
            let mut cells: HashMap<(usize, usize, usize), (Vec<u8>, Vec<u8>)> = HashMap::new();
            let mut write_pos: Vec<u32> = vec![0u32; nkv_clone.len()];
            for (layer, seed) in seeded_layers {
                let nkv_l = nkv_clone[layer];
                let cap_l = capacity_clone[layer];
                let hd_l = head_dim_clone[layer];
                let chunk = hd_l * kv_dtype_bytes;
                for h in 0..nkv_l {
                    for slot in 0..cap_l {
                        let mut k = vec![0u8; chunk];
                        let mut v = vec![0u8; chunk];
                        for (i, b) in k.iter_mut().enumerate() {
                            *b = seed
                                ^ (layer as u8)
                                ^ (h as u8).wrapping_mul(7)
                                ^ (slot as u8).wrapping_mul(13)
                                ^ (i as u8).wrapping_mul(3)
                                ^ 0x5A;
                        }
                        for (i, b) in v.iter_mut().enumerate() {
                            *b = seed
                                ^ (layer as u8)
                                ^ (h as u8).wrapping_mul(7)
                                ^ (slot as u8).wrapping_mul(13)
                                ^ (i as u8).wrapping_mul(3)
                                ^ 0xA5;
                        }
                        cells.insert((layer, h, slot), (k, v));
                    }
                }
            }
            while let Some(req) = rx.blocking_recv() {
                match req {
                    Request::Shutdown => break,
                    Request::KvSnapshot {
                        layer_rank,
                        range,
                        reply,
                    } => {
                        let result: Result<Option<KvSnapshotBytes>> =
                            if layer_rank >= nkv_clone.len() {
                                Err(anyhow::anyhow!("layer OOB"))
                            } else {
                                let nkv_l = nkv_clone[layer_rank];
                                let cap_l = capacity_clone[layer_rank];
                                let hd_l = head_dim_clone[layer_rank];
                                let is_sliding_l = is_sliding_clone[layer_rank];
                                let chunk = hd_l * kv_dtype_bytes;
                                // If layer was never seeded AND no
                                // restore wrote into it, return None
                                // (mirroring "no prefill yet").
                                let layer_populated =
                                    cells.iter().any(|((l, _, _), _)| *l == layer_rank);
                                if !layer_populated {
                                    Ok(None)
                                } else {
                                    let n_tokens = (range.end - range.start) as usize;
                                    let mut k_out = Vec::with_capacity(nkv_l * n_tokens * chunk);
                                    let mut v_out = Vec::with_capacity(nkv_l * n_tokens * chunk);
                                    for h in 0..nkv_l {
                                        for tok in range.start..range.end {
                                            let slot = if is_sliding_l {
                                                (tok as usize) % cap_l
                                            } else {
                                                tok as usize
                                            };
                                            match cells.get(&(layer_rank, h, slot)) {
                                                Some((k, v)) => {
                                                    k_out.extend_from_slice(k);
                                                    v_out.extend_from_slice(v);
                                                }
                                                None => {
                                                    k_out.extend_from_slice(&vec![0u8; chunk]);
                                                    v_out.extend_from_slice(&vec![0u8; chunk]);
                                                }
                                            }
                                        }
                                    }
                                    Ok(Some(KvSnapshotBytes {
                                        k: k_out,
                                        v: v_out,
                                        nkv_heads: nkv_l as u16,
                                        head_dim: hd_l as u16,
                                        capacity: cap_l as u32,
                                        is_sliding: is_sliding_l,
                                        write_pos: if is_sliding_l {
                                            write_pos[layer_rank]
                                        } else {
                                            u32::MAX
                                        },
                                    }))
                                }
                            };
                        let _ = reply.send(result);
                    }
                    Request::KvRestore {
                        layer_rank,
                        range,
                        k_payload,
                        v_payload,
                        write_pos: wp,
                        reply,
                    } => {
                        let result: Result<()> = if layer_rank >= nkv_clone.len() {
                            Err(anyhow::anyhow!("layer OOB"))
                        } else {
                            let nkv_l = nkv_clone[layer_rank];
                            let cap_l = capacity_clone[layer_rank];
                            let hd_l = head_dim_clone[layer_rank];
                            let is_sliding_l = is_sliding_clone[layer_rank];
                            let chunk = hd_l * kv_dtype_bytes;
                            let n_tokens = (range.end - range.start) as usize;
                            let expected = nkv_l * n_tokens * chunk;
                            if k_payload.len() != expected || v_payload.len() != expected {
                                Err(anyhow::anyhow!("payload size mismatch"))
                            } else {
                                let mut off = 0usize;
                                for h in 0..nkv_l {
                                    for tok in range.start..range.end {
                                        let slot = if is_sliding_l {
                                            (tok as usize) % cap_l
                                        } else {
                                            tok as usize
                                        };
                                        cells.insert(
                                            (layer_rank, h, slot),
                                            (
                                                k_payload[off..off + chunk].to_vec(),
                                                v_payload[off..off + chunk].to_vec(),
                                            ),
                                        );
                                        off += chunk;
                                    }
                                }
                                if is_sliding_l && wp != u32::MAX {
                                    write_pos[layer_rank] = wp;
                                }
                                Ok(())
                            }
                        };
                        let _ = reply.send(result);
                    }
                    _ => {
                        // Drop other request kinds — caller never
                        // awaits a reply for them in this fixture.
                    }
                }
            }
        })
        .expect("spawn synthetic kv-bridge worker");

    Engine {
        inner: Arc::new(EngineInner {
            tx,
            worker_handle: Mutex::new(Some(handle)),
            info: synthetic_load_info("synth-kv-bridge-test"),
            arch: LoadedArch::Gemma,
            model_id: "synth-kv-bridge-test".into(),
            context_length: None,
            quant_type: None,
            hidden_size: 0,
            vocab_size: 0,
            eos_token_ids: vec![],
            tokenizer: Arc::new(Tokenizer::new(tokenizers::models::bpe::BPE::default())),
            chat_template: Arc::new(String::new()),
            registration: None,
            token_bytes: std::sync::OnceLock::new(),
            kv_spill_descriptor: Some(descriptor_for_engine),
            tq_packed_descriptor: None,
            mode: EngineMode::SerialFifo,
            // ADR-040 C2b scaffold for synthetic test fixtures —
            // mirrors the production `Engine::spawn` shape; no live
            // worker drives the scheduler so the snapshot is a sentinel
            // FifoSerial-shape `SchedulerStats` with zero counters.
            max_slots: 1,
            // ADR-040 §3.5 iter-A5b: synthetic fixtures default to 0
            // (enforcement disabled — `synthetic_load_info` zeroes the
            // arch facts so `kv_bytes_per_token` returns 0 anyway).
            per_slot_kv_budget_bytes: 0,
            kv_bytes_per_token_cached: 0,
            scheduler_stats_snapshot: Arc::new(Mutex::new(SchedulerStats {
                policy: SchedulerPolicy::FifoSerial,
                in_flight_slots: 0,
                queue_capacity: 8,
                admitted_total: 0,
                rejected_429_total: 0,
                completed_total: 0,
            })),
        }),
    }
}

#[cfg(test)]
pub(crate) fn make_synthetic_engine_for_test(arch: LoadedArch) -> Engine {
    let (tx, mut rx) = mpsc::channel::<Request>(8);
    let handle = std::thread::Builder::new()
        .name("hf2q-engine-synthetic-test".into())
        .spawn(move || {
            while let Some(req) = rx.blocking_recv() {
                if matches!(req, Request::Shutdown) {
                    break;
                }
            }
        })
        .expect("spawn synthetic test worker");

    Engine {
        inner: Arc::new(EngineInner {
            tx,
            worker_handle: Mutex::new(Some(handle)),
            info: synthetic_load_info("iter-215-test-model"),
            arch,
            model_id: "iter-215-test-model".into(),
            context_length: None,
            quant_type: None,
            hidden_size: 0,
            vocab_size: 0,
            eos_token_ids: vec![],
            tokenizer: Arc::new(Tokenizer::new(tokenizers::models::bpe::BPE::default())),
            chat_template: Arc::new(String::new()),
            registration: None,
            token_bytes: std::sync::OnceLock::new(),
            kv_spill_descriptor: None,
            tq_packed_descriptor: None,
            mode: EngineMode::SerialFifo,
            // ADR-040 C2b scaffold for synthetic test fixtures —
            // mirrors production shape with zero-counter sentinel.
            max_slots: 1,
            // ADR-040 §3.5 iter-A5b: synthetic fixtures default to 0
            // (enforcement disabled).
            per_slot_kv_budget_bytes: 0,
            kv_bytes_per_token_cached: 0,
            scheduler_stats_snapshot: Arc::new(Mutex::new(SchedulerStats {
                policy: SchedulerPolicy::FifoSerial,
                in_flight_slots: 0,
                queue_capacity: 8,
                admitted_total: 0,
                rejected_429_total: 0,
                completed_total: 0,
            })),
        }),
    }
}

/// **ADR-040 §6.1.18 iter-A5d (Critical #2, 3rd reaffirmation closure)** —
/// synthetic `Engine` for the iter-A5d **streaming** handler-level test in
/// `src/serve/api/handlers.rs`.
///
/// Carries a non-zero `per_slot_kv_budget_bytes` + `kv_bytes_per_token_cached`
/// so that `Engine::try_admit_budget(prompt, max_tokens)` actually surfaces
/// `EngineAdmitError::SlotBudgetExceeded`. The worker drains until shutdown
/// — the streaming handler's pre-stream admit at `handlers.rs:1748` returns
/// early on the over-budget error, so the worker is never reached.
///
/// Signature uses only public/`pub(crate)` types (`LoadedArch` + `Engine`)
/// so the helper is callable from `handlers.rs` tests without leaking the
/// private `Request` enum out of `engine.rs`.
#[cfg(test)]
pub(crate) fn make_synthetic_engine_over_budget(
    arch: LoadedArch,
    per_slot_kv_budget_bytes: u64,
    kv_bytes_per_token_cached: u64,
) -> Engine {
    let (tx, mut rx) = mpsc::channel::<Request>(8);
    let handle = std::thread::Builder::new()
        .name("hf2q-engine-a5d-stream-test".into())
        .spawn(move || {
            // Drain-until-shutdown — streaming pre-admit short-circuits
            // BEFORE any request reaches the worker (handlers.rs:1748);
            // this worker exists only so `Engine::shutdown()` can join.
            while let Some(req) = rx.blocking_recv() {
                if matches!(req, Request::Shutdown) {
                    break;
                }
            }
        })
        .expect("spawn a5d streaming-handler-test worker");

    Engine {
        inner: Arc::new(EngineInner {
            tx,
            worker_handle: Mutex::new(Some(handle)),
            info: synthetic_load_info("a5d-over-budget-test-model"),
            arch,
            model_id: "a5d-over-budget-test-model".into(),
            context_length: None,
            quant_type: None,
            hidden_size: 0,
            vocab_size: 0,
            eos_token_ids: vec![],
            tokenizer: Arc::new(Tokenizer::new(tokenizers::models::bpe::BPE::default())),
            chat_template: Arc::new(String::new()),
            registration: None,
            token_bytes: std::sync::OnceLock::new(),
            kv_spill_descriptor: None,
            tq_packed_descriptor: None,
            mode: EngineMode::SerialFifo,
            max_slots: 1,
            per_slot_kv_budget_bytes,
            kv_bytes_per_token_cached,
            scheduler_stats_snapshot: Arc::new(Mutex::new(SchedulerStats {
                policy: SchedulerPolicy::FifoSerial,
                in_flight_slots: 0,
                queue_capacity: 8,
                admitted_total: 0,
                rejected_429_total: 0,
                completed_total: 0,
            })),
        }),
    }
}

/// **ADR-040 §6.1.18 iter-A5d** — synthetic `Engine` for the iter-A5d
/// **non-streaming** handler-level test in `src/serve/api/handlers.rs`.
///
/// The worker thread responds to a `Request::Generate` with a pre-canned
/// `Err("slot_budget_exceeded: ...needed_bytes=N, budget_bytes=B...")`
/// matching the EXACT format the production `worker_run` emits at
/// `engine.rs:3834-3839`. The handler's string-match arm at
/// `handlers.rs::chat_completions_with_prepared` (line ~447) then routes
/// the error to `ApiError::slot_budget_exceeded(N, B).into_response()` —
/// which is exactly what the test asserts.
///
/// `per_slot_kv_budget_bytes` + `kv_bytes_per_token_cached` are both set
/// to 0 because the non-streaming path does NOT call `try_admit_budget`
/// (only the streaming path does — handlers.rs:1748); the worker reply
/// IS the path under test for non-streaming.
///
/// Signature uses only public/`pub(crate)` types (`LoadedArch` + `Engine`)
/// so the helper is callable from `handlers.rs` tests without leaking
/// the private `Request` enum out of `engine.rs`.
#[cfg(test)]
pub(crate) fn make_synthetic_engine_with_slot_budget_exceeded_worker(
    arch: LoadedArch,
    needed_bytes: u64,
    budget_bytes: u64,
) -> Engine {
    let (tx, mut rx) = mpsc::channel::<Request>(8);
    let handle = std::thread::Builder::new()
        .name("hf2q-engine-a5d-nonstream-test".into())
        .spawn(move || {
            while let Some(req) = rx.blocking_recv() {
                match req {
                    Request::Shutdown => break,
                    Request::Generate { reply, .. } => {
                        let _ = reply.send(Err(anyhow::anyhow!(
                            "slot_budget_exceeded: ADR-040 §3.5 A5b — per-slot \
                             KV budget exceeded (needed_bytes={}, budget_bytes={}). \
                             Reduce max_tokens or use a shorter prompt.",
                            needed_bytes,
                            budget_bytes
                        )));
                    }
                    _ => {
                        // Drop other request kinds — the non-streaming
                        // handler test only ever sends a Generate.
                    }
                }
            }
        })
        .expect("spawn a5d non-streaming-handler-test worker");

    Engine {
        inner: Arc::new(EngineInner {
            tx,
            worker_handle: Mutex::new(Some(handle)),
            info: synthetic_load_info("a5d-worker-error-test-model"),
            arch,
            model_id: "a5d-worker-error-test-model".into(),
            context_length: None,
            quant_type: None,
            hidden_size: 0,
            vocab_size: 0,
            eos_token_ids: vec![],
            tokenizer: Arc::new(Tokenizer::new(tokenizers::models::bpe::BPE::default())),
            chat_template: Arc::new(String::new()),
            registration: None,
            token_bytes: std::sync::OnceLock::new(),
            kv_spill_descriptor: None,
            tq_packed_descriptor: None,
            mode: EngineMode::SerialFifo,
            max_slots: 1,
            // Non-streaming path does NOT call try_admit_budget; the
            // error comes from the worker reply (see comment above).
            per_slot_kv_budget_bytes: 0,
            kv_bytes_per_token_cached: 0,
            scheduler_stats_snapshot: Arc::new(Mutex::new(SchedulerStats {
                policy: SchedulerPolicy::FifoSerial,
                in_flight_slots: 0,
                queue_capacity: 8,
                admitted_total: 0,
                rejected_429_total: 0,
                completed_total: 0,
            })),
        }),
    }
}

struct EngineInner {
    tx: mpsc::Sender<Request>,
    /// Worker-thread join handle. Held in a `Mutex<Option<...>>` so
    /// `Engine::shutdown` can `take()` it once and `.join()` the thread, and
    /// callers can be cheap-clone an `Engine` without contending on the
    /// handle. Outside of shutdown, the slot is read-only.
    worker_handle: Mutex<Option<JoinHandle<()>>>,
    /// Unified load snapshot built once at model-load completion. Serve
    /// startup reads this for the ADR-018 banner and tracing without
    /// touching the worker thread.
    info: Arc<LoadInfo>,
    /// Iter-215 Wedge-2: which `LoadedModel` variant the worker
    /// thread owns.  Cached at spawn time so handlers can dispatch on
    /// the architecture without round-tripping a request.  Drives the
    /// HTTP 501 short-circuit on chat / embed / vision endpoints for
    /// the `Qwen35` variant; `Gemma` is unconstrained (production
    /// path).  Wedge-3 (full Qwen3.5/3.6 inference) flips the Qwen35
    /// arm from 501 to live.
    arch: LoadedArch,
    /// Metadata exposed to handlers without touching the worker thread.
    /// Immutable for the lifetime of the engine.
    model_id: String,
    context_length: Option<usize>,
    quant_type: Option<String>,
    /// Hidden-state dimensionality of the loaded model.  Surfaced to the
    /// `/v1/embeddings` handler when the chat model is used as an
    /// embedder (Phase 2a Task #8) — used to validate the OpenAI
    /// `dimensions` parameter and to size the response payload.
    hidden_size: usize,
    /// Vocabulary size — needed to size the per-vocab token_bytes table
    /// (built lazily on first grammar request).
    vocab_size: usize,
    eos_token_ids: Vec<u32>,
    /// Tokenizer cloned per-request so handlers can tokenize without a lock.
    tokenizer: Arc<Tokenizer>,
    /// Chat-template string (GGUF metadata or fallback). Rendered per request
    /// using `minijinja`.
    chat_template: Arc<String>,
    /// Per-model registration — reasoning-boundary + tool-call markers
    /// (Decision #21). `None` when no family matches this model's id.
    registration: Option<super::registry::ModelRegistration>,
    /// Per-vocab decoded UTF-8 byte table — `token_bytes[id]` is the
    /// bytes the tokenizer emits when token `id` is sampled.  Built on
    /// first grammar request via `Engine::token_bytes_table()` and
    /// cached for the engine's lifetime (vocab × ~3-5 bytes ≈ 1 MB at
    /// 256K vocab — trivial vs the model weights).  `OnceLock` so
    /// concurrent first-callers race only on the build, not on reads.
    /// Phase 2a Task #5 / iter-95.
    token_bytes: std::sync::OnceLock<Arc<Vec<Vec<u8>>>>,
    /// Phase B-dense.2 follow-up: cached KV-spill shape descriptor for
    /// the Gemma 4 dense F32/F16 K/V cache. Populated at `Engine::spawn`
    /// time from the `GemmaLoadedModel`'s `MlxModelWeights` BEFORE the
    /// model is moved into the worker thread. `None` for the `Qwen35`
    /// variant (its KV state is hybrid DeltaNet + full-attention and
    /// belongs to a future B-hybrid descriptor).
    ///
    /// Read-only after construction — exposed via
    /// [`Engine::kv_spill_descriptor`] so the
    /// `Gemma4DenseSpillFactory::try_from_engine_arc` can build a
    /// real (non-stub) hook from the live shape without round-tripping
    /// through the worker channel.
    kv_spill_descriptor: Option<super::kv_spill_descriptor::KvSpillDescriptor>,

    /// **ADR-017 §B-tq.4 iter-4** — per-layer TQ-packed runtime
    /// shape captured at engine spawn (mirrors `kv_spill_descriptor`
    /// for the dense path).  Populated when `HF2Q_TQ_KV=1` AND the
    /// loaded model is Gemma 4.  `None` otherwise.
    ///
    /// The factory's `try_construct` reads this to build a per-
    /// layer-correct `TqPackedConfig` instead of the
    /// `cmd_serve` fallback (which was a 2-layer
    /// [sliding, global] cfg that mis-shapes Gemma 4's actual
    /// layer pattern and caused `restore_block CodecErr` bails on
    /// `layer >= 1`).
    tq_packed_descriptor: Option<super::tq_packed_descriptor::TqPackedSpillDescriptor>,

    /// **ADR-040 Phase C iter-1.5** — the scheduling mode this engine was
    /// constructed under. Populated by [`Engine::spawn`] (always
    /// [`EngineMode::SerialFifo`]) and by [`Engine::spawn_with_mode`]
    /// (echoes the caller's requested mode after validation).
    ///
    /// Stored even when the runtime behaviour is still `SerialFifo` so
    /// that [`Engine::mode`] does NOT lie about what was requested — the
    /// Liskov-substitution honest version. Iter-1's "always return
    /// `SerialFifo::default()`" was flagged as a critical violation by
    /// both adversarial reviewers (Codex + Claude); iter-1.5 ships the
    /// stored field + fail-fast rejection of unwired modes.
    ///
    /// At iter-1.5 the only mode that survives validation in
    /// `spawn_with_mode` is `SerialFifo`; `SlotAware` is rejected with
    /// [`EngineSpawnError::ModeNotYetWired`] at the API boundary.
    mode: EngineMode,

    /// **ADR-040 Phase C iter-2a (C2b) — slot cap snapshot** (dossier
    /// `docs/research/adr040-c2-wiring-dossier-2026-05-24.md` §2.2).
    ///
    /// Populated at spawn time from [`EngineMode`]:
    /// - `SerialFifo` ⇒ `1` (ADR-005 Phase 2 single-in-flight contract).
    /// - `SlotAware { max_slots }` ⇒ `max_slots` (gated at
    ///   `spawn_with_mode` until iter-2b lifts the rejection).
    ///
    /// Read by future `/metrics` / `/v1/models` extensions (Phase C3).
    /// Iter-2a only positions the field; the SlotAware runtime that
    /// actually exercises `max_slots > 1` lands at iter-2b/2c.
    max_slots: u32,
    /// **ADR-040 §3.5 iter-A5b** — per-slot KV byte budget. Computed
    /// at spawn time as `kv_cache_budget_bytes / max_slots` (`0`
    /// means enforcement disabled, preserving pre-A5
    /// byte-equivalence). Read by [`Engine::try_admit_budget`] +
    /// surfaced to the worker thread for the scheduler-side
    /// `FifoSchedulerAdapter::new_with_kv_budget`/
    /// `InflightBatchedScheduler::new_with_kv_budget` constructor.
    ///
    /// `0` semantics match
    /// [`crate::serve::scheduler::FifoSchedulerAdapter::per_slot_kv_budget_bytes`]:
    /// no enforcement at any admit path. Under SerialFifo this is
    /// `kv_cache_budget_bytes.unwrap_or(0) / 1` so an operator who
    /// does NOT pass `--kv-cache-budget-bytes` retains the pre-A5
    /// "unbounded" semantics verbatim.
    per_slot_kv_budget_bytes: u64,
    /// **ADR-040 §3.5 iter-A5b** — cached per-token KV byte cost
    /// (`LoadInfo::kv_bytes_per_token()` value captured at spawn).
    /// `0` ⇒ unknown / synthetic-fixture loader; admit-side check
    /// short-circuits to "do not enforce" (same opt-out as
    /// `per_slot_kv_budget_bytes == 0`).
    kv_bytes_per_token_cached: u64,
    /// **ADR-040 Phase C iter-2a (C2b) — scheduler stats snapshot**
    /// (dossier §2.2). The actual scheduler lives on the worker thread
    /// (Shape A); after each `release` the worker writes a snapshot of
    /// [`SchedulerStats`] here so handler-side `/metrics` reads pay only
    /// a brief `Mutex` acquisition (no cross-thread scheduler access).
    /// Phase C3 wires this into the Prometheus exposition.
    scheduler_stats_snapshot: Arc<Mutex<SchedulerStats>>,
}

#[cfg(test)]
fn synthetic_load_info(model_id: &str) -> Arc<LoadInfo> {
    Arc::new(LoadInfo {
        model_id: model_id.to_string(),
        arch_str: "gemma4".to_string(),
        arch_family: ArchFamily::Gemma4,
        model_path: PathBuf::from(format!("/tmp/{model_id}.gguf")),
        on_disk_bytes: 0,
        backend_chip: "test-gpu".to_string(),
        backend: "mlx-native",
        n_layers: 0,
        hidden_size: 0,
        vocab_size: 0,
        n_attention_heads: 0,
        n_key_value_heads: 0,
        head_dim: 0,
        sliding_window: None,
        full_attention_interval: None,
        max_context_length: None,
        moe: None,
        quant_label: None,
        quant_bpw: None,
        tokenizer_source: TokenizerSource::GgufEmbedded,
        eos_token_ids: Vec::new(),
        bos_token_id: None,
        chat_template_source: ChatTemplateSource::None,
        provenance: crate::core::provenance::Provenance::External,
        vision_projector: None,
        load_wall_clock: Duration::ZERO,
        resident_weight_bytes: None,
        kv_cache_budget_bytes: None,
        kv_spill_active: false,
        tq_kv_active: false,
        kv_bytes_per_token_override: None,
    })
}

/// The request protocol the worker thread drains.
enum Request {
    Warmup {
        reply: oneshot::Sender<Result<()>>,
    },
    Generate {
        prompt_tokens: Vec<u32>,
        params: SamplingParams,
        reply: oneshot::Sender<Result<GenerationResult>>,
    },
    /// Streaming generation — tokens flow back to the handler via `events`
    /// as `GenerationEvent::Delta{ kind, text }` per decode step, then a
    /// terminating `Done { finish_reason, prompt_tokens, completion_tokens,
    /// stats }` (or `Error`). When the handler's SSE stream is dropped
    /// (client disconnect per Decision #18), `events.send` returns Err and
    /// the worker breaks early — the queue slot is freed immediately.
    /// `cancellation_counter` (if `Some`) is incremented by 1 when the
    /// worker aborts because the receiver was dropped; surfaced via
    /// `hf2q_sse_cancellations` in `/metrics`.
    GenerateStream {
        prompt_tokens: Vec<u32>,
        params: SamplingParams,
        events: mpsc::Sender<super::sse::GenerationEvent>,
        cancellation_counter: Option<Arc<std::sync::atomic::AtomicU64>>,
        /// Per-position embedding overrides for the multimodal vision
        /// path (Phase 2c iter-211 W79). Empty slice ⇒ identity over
        /// the text-only `forward_prefill` path (the prefill function
        /// is already a thin wrapper around
        /// `forward_prefill_with_soft_tokens` with an empty slice —
        /// see `src/serve/forward_prefill.rs:111-118`).
        ///
        /// Pre-iter-211 the streaming worker did not carry soft-token
        /// data and the chat handler returned a 400 when an `image_url`
        /// content part was present + `stream: true`. Iter-211 closes
        /// AC 3103 by routing soft tokens through the streaming path
        /// using the same forward-prefill API the non-streaming
        /// `Request::GenerateWithSoftTokens` arm already uses.
        soft_tokens: Vec<SoftTokenData>,
        /// **Wedge-4e (iter-224 row 5)**: DeepStack injection chunks
        /// split out from the augmented `[n_image_tokens, hidden *
        /// (1 + N_deepstack)]` ViT output. `None` for Gemma /
        /// non-Qwen3-VL paths and for the legacy text-only / pure
        /// soft-token streaming requests; the Qwen35 streaming arm
        /// rebuilds borrowed `DeepstackInjection<'_>` slices from this
        /// owned `DeepstackData` for the per-token forward pass.
        ///
        /// Mirrors the `Request::GenerateWithSoftTokens.deepstack`
        /// field added in Wedge-4d so the streaming and non-streaming
        /// paths consume the same engine-seam shape.
        deepstack: Option<DeepstackData>,
        /// **Wedge-4e (iter-224 row 5)**: 3D-mRoPE position buffer
        /// (`[4 * prompt_len]` axis-major i32) built by
        /// `build_qwen3vl_positions`. `None` for Gemma / non-Qwen3-VL
        /// streaming paths (the Qwen35 stream worker arm synthesizes
        /// `prefill_positions_for(prompt_len)` text-style positions
        /// when this is None).
        ///
        /// Mirrors the `Request::GenerateWithSoftTokens.positions_flat`
        /// field added in Wedge-4d.
        positions_flat: Option<Vec<i32>>,
    },
    /// Pooled-embedding request (ADR-005 Phase 2a Task #8 / iter-92).
    ///
    /// Runs the chat model's prefill forward pass and returns the
    /// L2-normalized last-token hidden state — the natural "Last" pooling
    /// for autoregressive (causal-attention) chat models.  Used by the
    /// `/v1/embeddings` handler when no `--embedding-model` is loaded but
    /// the chat model is.  See
    /// `MlxModelWeights::forward_embed_last` for the GPU-side semantics.
    Embed {
        prompt_tokens: Vec<u32>,
        reply: oneshot::Sender<Result<Vec<f32>>>,
    },
    /// Vision-aware chat generation (Phase 2c Task #17 / iter-98).
    /// See `Engine::generate_with_soft_tokens` doc.
    ///
    /// **iter-224 Wedge-4d**: extended with an optional
    /// `deepstack: Option<DeepstackData>` and an optional
    /// `positions_flat: Option<Vec<i32>>` for the Qwen3-VL path.
    /// Gemma + ClipClassic paths leave both `None` (Gemma's text-mode
    /// IMROPE positions are synthesized inside the worker; Gemma has
    /// no deepstack heads).
    GenerateWithSoftTokens {
        prompt_tokens: Vec<u32>,
        soft_tokens: Vec<SoftTokenData>,
        params: SamplingParams,
        /// Wedge-4d: DeepStack injection chunks split out from the
        /// augmented `[n_image_tokens, hidden * (1 + N_deepstack)]`
        /// ViT output. `None` for Gemma / non-Qwen3-VL paths.
        deepstack: Option<DeepstackData>,
        /// Wedge-4d: 3D-mRoPE position buffer
        /// (`[4 * prompt_len]` axis-major i32) built by
        /// `build_qwen3vl_positions`. `None` for Gemma / non-Qwen3-VL
        /// paths (the Qwen35 worker arm synthesizes
        /// `prefill_positions_for(prompt_len)` text-style positions
        /// when this is None).
        positions_flat: Option<Vec<i32>>,
        reply: oneshot::Sender<Result<GenerationResult>>,
    },
    /// Phase B-dense.2 follow-up: read a slice of `dense_kvs[layer_rank]`
    /// for the given token-position range. The worker pauses inference
    /// (no concurrent decode — the channel is FIFO-serial) and reads
    /// the K and V buffer bytes directly via `MlxBuffer::as_slice::<u8>()`.
    /// Returns `None` when `dense_kvs` is `None` (no prefill yet) or when
    /// the layer index is out of range; returns the concatenated K+V bytes
    /// otherwise (head-major order: `[nkv_heads, capacity, head_dim]`).
    ///
    /// The KV-persist hook calls this from
    /// `Gemma4DenseSpill::snapshot_block` via
    /// `Engine::request_kv_snapshot`. Only the Gemma variant honours the
    /// request; the Qwen35 arm returns `Ok(None)`.
    KvSnapshot {
        layer_rank: usize,
        range: std::ops::Range<u32>,
        reply: oneshot::Sender<Result<Option<KvSnapshotBytes>>>,
    },
    /// Phase B-dense.2 follow-up: write a slice of `dense_kvs[layer_rank]`
    /// for the given token-position range from previously-snapshotted
    /// bytes. The worker pauses inference, allocates `dense_kvs` if
    /// `None` (mirroring `forward_prefill.rs:274-285`), writes K and V
    /// bytes back via `MlxBuffer::as_mut_slice::<u8>()`. Returns
    /// `Err(...)` on shape mismatch or allocation failure.
    KvRestore {
        layer_rank: usize,
        range: std::ops::Range<u32>,
        k_payload: Vec<u8>,
        v_payload: Vec<u8>,
        /// Sliding ring write position to restore after the write
        /// (`u32::MAX` for full-attention layers — sentinel).
        write_pos: u32,
        reply: oneshot::Sender<Result<()>>,
    },
    /// **Phase B-tq.4** — TQ-packed K/V snapshot.  Worker thread
    /// reads `MlxModelWeights.kv_caches[layer]` via
    /// [`crate::inference::models::gemma4::MlxModelWeights::tq_v2_snapshot_block`]
    /// and returns `(k_payload, v_payload)` — two
    /// `tq_packed_v2` envelopes.  Mirror of [`Request::KvSnapshot`]
    /// for the TurboQuant-active KV path; Qwen35/Qwen3VL arms return
    /// `Err` (TQ is Gemma-4-only at this iter, per the
    /// family-scoping discipline).
    TqPackedKvSnapshot {
        layer_rank: usize,
        range: std::ops::Range<u32>,
        bits_per_coord: crate::serve::kv_persist::families::tq_packed::TqBitsPerCoord,
        flags: u32,
        scale: f64,
        reply: oneshot::Sender<Result<(Vec<u8>, Vec<u8>)>>,
    },
    /// **Phase B-tq.4** — TQ-packed K/V restore.  Worker thread
    /// writes `(k_payload, v_payload)` into
    /// `MlxModelWeights.kv_caches[layer]` via
    /// [`crate::inference::models::gemma4::MlxModelWeights::tq_v2_restore_block`].
    TqPackedKvRestore {
        layer_rank: usize,
        range: std::ops::Range<u32>,
        bits_per_coord: crate::serve::kv_persist::families::tq_packed::TqBitsPerCoord,
        k_payload: Vec<u8>,
        v_payload: Vec<u8>,
        reply: oneshot::Sender<Result<()>>,
    },
    /// ADR-017 Closure iter-5 / Phase E (2026-05-04) — serialize the
    /// worker-side `loaded.prompt_cache` into a JSON byte payload via
    /// [`crate::serve::kv_persist::prompt_cache_persist::try_serialize`].
    /// Returns `Ok(None)` when the cache is empty or grammar-bound
    /// (see module docs). The worker is the sole owner of the cache,
    /// so this is the only race-free path to read it.
    PromptCacheSnapshot {
        reply: oneshot::Sender<Result<Option<Vec<u8>>>>,
    },
    /// ADR-017 Closure iter-5 / Phase E (2026-05-04) — restore the
    /// worker-side `loaded.prompt_cache` from a JSON byte payload via
    /// [`crate::serve::kv_persist::prompt_cache_persist::try_deserialize`].
    /// Returns `Err(...)` on parse failure or schema-version mismatch.
    /// Caller is expected to fire this BEFORE the first request
    /// arrives at the freshly-loaded model.
    PromptCacheRestore {
        payload: Vec<u8>,
        reply: oneshot::Sender<Result<()>>,
    },
    /// Graceful-shutdown sentinel.
    Shutdown,
}

/// Phase B-dense.2 follow-up — return shape of the worker's
/// `KvSnapshot` arm. K and V bytes are returned as separate buffers
/// because the per-family payload codec (`gemma4_dense.rs`) keeps them
/// in different envelope sections — copying the worker-side memcpy
/// directly into the codec's two output slots avoids one extra split.
#[derive(Debug, Clone)]
pub struct KvSnapshotBytes {
    /// K bytes, head-major: `[nkv_heads, n_tokens, head_dim]`.
    pub k: Vec<u8>,
    /// V bytes, head-major: `[nkv_heads, n_tokens, head_dim]`.
    pub v: Vec<u8>,
    /// Per-layer shape captured at the worker side so the caller can
    /// validate against its descriptor without re-reading.
    pub nkv_heads: u16,
    pub head_dim: u16,
    pub capacity: u32,
    pub is_sliding: bool,
    /// Sliding ring write position (or `u32::MAX` sentinel for
    /// full-attention layers).
    pub write_pos: u32,
}

// ---------------------------------------------------------------------------
// Load path — LoadedModel
// ---------------------------------------------------------------------------

/// All the artifacts needed for inference, held together so the worker can
/// take ownership in a single move.
///
/// ADR-005 Phase 4 reopen iter-215 Wedge-2: previously a flat struct
/// targeting only the Gemma-shaped `forward_mlx` path.  The struct →
/// enum lift here adds a `Qwen35` variant so the SERVE-side load path
/// can dispatch on `general.architecture` (replacing iter-214's
/// `load_engine` arch-detect bail with actual Qwen3.5/3.6 model load).
///
/// Inference for the `Qwen35` variant is OUT OF iter-215 MVP scope —
/// the worker arm returns HTTP 501 with an operator-actionable message
/// pointing at `hf2q generate` (cmd_generate_qwen35) for chat today.
/// Wedge-3 (deferred follow-up) wires `Qwen35Model::forward_*` into
/// the worker thread for full chat completion parity.
pub enum LoadedModel {
    /// Gemma 4 (and Gemma-shaped) GGUFs.  Drives the production
    /// `forward_mlx` chat-completion path.
    Gemma(GemmaLoadedModel),
    /// Qwen3.5 / Qwen3.6 (dense + MoE) GGUFs.  Loaded via
    /// `Qwen35Model::load_from_gguf`.  Inference path returns 501 in
    /// iter-215 MVP; Wedge-3 wires forward_gpu through the worker.
    Qwen35(super::engine_qwen35::Qwen35LoadedModel),
    /// Qwen3-VL text-LM GGUFs (ADR-005 Wedge-4 / iter-228a). Loaded via
    /// [`crate::inference::models::qwen3vl_text::Qwen3VlTextModel::load_from_gguf`].
    /// Inference path returns 501 in iter-228a MVP via the
    /// [`crate::inference::models::qwen3vl_text::forward::QWEN3VL_TEXT_FORWARD_PENDING_SENTINEL`]
    /// sentinel; iter-228b wires the dense transformer forward.
    ///
    /// Replaces iter-227's actionable-error bail at the dispatch site
    /// (the GGUF now opens cleanly and the model loads, but the chat
    /// arm short-circuits to 501 the same way Qwen35 did at iter-215).
    Qwen3VlText(super::engine_qwen3vl::Qwen3VlTextLoadedModel),
    /// DeepSeek-V4-Flash native verifier + persistent appendable cache.
    Deepseek4(super::engine_deepseek4::Deepseek4LoadedModel),
}

/// Gemma 4 (and Gemma-shaped) artifacts.  Pre-iter-215 these were the
/// fields of `LoadedModel` directly; iter-215 nests them inside the
/// enum's `Gemma` variant so a sibling `Qwen35` variant can land
/// without breaking call sites.
pub struct GemmaLoadedModel {
    pub weights: MlxModelWeights,
    pub ctx: GpuContext,
    pub config: Gemma4Config,
    pub model_id: String,
    pub model_path: PathBuf,
    pub tokenizer_path: PathBuf,
    pub context_length: Option<usize>,
    pub quant_type: Option<String>,
    pub tokenizer: Tokenizer,
    pub chat_template: String,
    pub eos_token_ids: Vec<u32>,
    pub load_duration: Duration,
    /// Single-slot prompt cache (Phase 2a Task #7 / Decision #24, iter-96).
    /// Owned by the worker thread; lives across requests.  See
    /// `PromptCache` doc for the cache contract.
    pub prompt_cache: PromptCache,
    /// ADR-017 Phase E option (a) iter-2 — LCP partial-prefix
    /// observability registry. Detects whether the current request's
    /// prompt shares a non-trivial prefix with a previously-served
    /// prompt under the same `(model_fingerprint, tenant_id,
    /// params_hash)` tuple. Iter-2 ships **detection only**: hits
    /// bump `KvSpillCounters::lcp_*` counters but the partial-prefill
    /// resume path stays OFF — `forward_prefill` still resets
    /// `write_pos = 0` per its iter-1 contract. Iter-3 (highest-risk;
    /// requires Codex Phase-2b audit per memory
    /// `feedback_codex_review_catches_unified_memory_races`) flips the
    /// payload to `Vec<Arc<DenseKvBuffers>>` and conditionally honors
    /// the cached prefix.
    ///
    /// Capacity = 16 entries — covers /cfa Phase 2 fan-out (≤8
    /// workers sharing one system prompt) and multi-turn chat (last
    /// 16 turns visible). Iter-3 may make this env-tunable via
    /// `HF2Q_KV_LCP_CAPACITY`. Iter-3 swaps the marker payload `()`
    /// out for `crate::inference::models::gemma4::DenseKvBuffers`: the
    /// registry now stores per-layer Arc clones of the actual
    /// post-prefill KV state, ready for in-place reuse on a
    /// partial-prefix hit when `HF2Q_KV_LCP_RESUME=1` (default OFF).
    /// "gemma-hybrid-lcp" (2026-08-03): payload is now the
    /// regime-aware `GemmaLcpLayerKv` enum — `Dense` under
    /// `HF2Q_USE_DENSE=1`, `DenseAndHybrid` under the production
    /// hybrid regime (dense leg for prefill SDPA + hybrid leg for
    /// decode). Restoring both legs is what makes LCP resume coherent
    /// in production; see the enum's doc in `gemma4/kv_cache.rs`.
    pub lcp_registry: crate::serve::kv_persist::lcp_registry::LcpRegistry<
        crate::inference::models::gemma4::GemmaLcpLayerKv,
    >,
    /// ADR-017 Phase E.a iter-2 — handle to the AppState-owned
    /// `KvSpillCounters` so per-request LCP probes bump the same Arc
    /// the `/metrics` handler reads. `None` for tests / standalone
    /// engine constructions / Qwen35 path (whose worker arm
    /// short-circuits to 501 before any LCP probe could fire).
    /// Set by `serve::load_engine` from `EngineConfig.kv_metrics_sink`
    /// BEFORE `Engine::spawn` moves the loaded model into the worker.
    pub kv_metrics_sink:
        Option<std::sync::Arc<dyn crate::serve::kv_persist::metrics::KvCacheMetricsSink>>,
    /// ADR-017 §F4 — GGUF provenance captured at load time via
    /// `crate::core::provenance::detect(&gguf)`. Threaded into the
    /// `KvSpillDescriptor` at `Engine::spawn` so the per-family hook
    /// (Phase B-dense.2) can build a strict `ModelFingerprint`
    /// namespace key for hf2q-quantized GGUFs and fall back to the
    /// legacy `(repo, quant)` key for foreign GGUFs (`Provenance::External`).
    /// Read once at spawn; not consulted afterwards.
    pub provenance: crate::core::provenance::Provenance,

    /// **ADR-040 Phase C iter-2c (C2c)** — multi-seq KV scaffolds
    /// provisioned at `spawn_with_mode(SlotAware { max_slots: N })` time
    /// using the A3a (`MultiSeqHbKvBuffers`) / A3b
    /// (`MultiSeqHybridKvBuffers`) per-layer allocators with
    /// `n_seqs = max_slots`.
    ///
    /// `None` under [`EngineMode::SerialFifo`] (preserves byte-equivalence
    /// vs. pre-ADR-040 — the legacy single-seq `MlxKvCache` on
    /// `MlxModelWeights.kv_caches` remains the live KV state for slot 0).
    /// `Some(_)` under [`EngineMode::SlotAware`]; one entry per layer
    /// matching `weights.layers.len()`.
    ///
    /// **Iter-C2c (Path B) scope**: provisioned structurally so the
    /// A3a/A3b allocators are exercised end-to-end at production shapes
    /// with `n_seqs = max_slots`, **but** the per-request forward path
    /// (`forward_prefill` + `forward_decode`) still reads/writes the
    /// legacy single-seq cache at SlotId(0). Slot N>0 admission surfaces
    /// `MultiSeqError::CapabilityUnsupported { capability:
    /// "gemma4-forward-prefill-slot-N (iter-C2c-cont)" }` at the worker
    /// arm — kernel-level slot-offset routing through these scaffolds is
    /// **iter-C2c-cont** scope (gated on Phase B4c, per ADR-040 §6
    /// + §6.1.21).
    ///
    /// **Variant choice**: the dense path uses
    /// [`crate::inference::models::gemma4::kv_cache::MultiSeqHbKvBuffers`]
    /// (TQ-packed HB KV) for both sliding and full-attention layers —
    /// matches the production `MlxKvCache` allocator at
    /// `gemma4/model.rs:1247-1301` (TurboQuant 4-bit nibble-packed
    /// indices + F32 norms). The A3b
    /// `MultiSeqHybridKvBuffers` (F16-K + TQ-HB-V or F16-V) is the
    /// path engaged when `HF2Q_FULL_F16_KV=1` or
    /// `HF2Q_DFLASH_XLEN_SDPA=1`; **iter-C2c (Path B)** picks the
    /// HB variant only (matches default operator config); the hybrid
    /// variant scaffold is iter-C2c-cont per the same gating as the
    /// kernel slot routing.
    pub multi_seq_kv: Option<Vec<crate::inference::models::gemma4::kv_cache::MultiSeqHbKvBuffers>>,

    /// **ADR-040 Phase C iter-C2c-cont (2026-05-30)** — sibling multi-
    /// seq KV scaffold for the PRODUCTION-DEFAULT hybrid F16-K + TQ-HB-V
    /// (or full F16) KV path.
    ///
    /// Provisioned at `spawn_with_mode(SlotAware { max_slots: N })` time
    /// using the A3b `alloc_multi_seq_hybrid_kv_for_layer` per-layer
    /// allocator with `n_seqs = max_slots`. Coexists with
    /// [`Self::multi_seq_kv`] (the HbKvBuffers HB-encoded opt-out
    /// scaffold C2c §6.1.21 provisions verbatim) — both can be `Some(_)`
    /// after a SlotAware spawn under default env (where
    /// `INVESTIGATION_ENV.hybrid_kv == true` per H10 falsification at
    /// §6.1.11 / ADR-029 iter-13).
    ///
    /// **Why both coexist (H91 hypothesis)**: the per-request KV regime
    /// is selected INSIDE the model fn body via
    /// [`crate::debug::INVESTIGATION_ENV.hybrid_kv`] read at call time
    /// (see `forward_prefill_with_soft_tokens_slot_aware`'s dispatch
    /// fork at `src/serve/forward_prefill.rs:2562`). Operators can flip
    /// the env at process start; both scaffolds must be available so
    /// the dispatch fork hands the appropriate one to the kernel. A
    /// unified enum wrapping would force a spawn-time regime
    /// commitment, breaking iter-2A's per-call selection contract.
    ///
    /// **Provisioning gate (H92 hypothesis)**:
    /// - `HF2Q_HYBRID_KV=1` (DEFAULT since ADR-029 iter-13): this field
    ///   is `Some(_)` after SlotAware spawn.
    /// - `HF2Q_HYBRID_KV=0` (opt-out): this field stays `None` (the HB-
    ///   encoded fallback is provisioned via [`Self::multi_seq_kv`]).
    /// - [`EngineMode::SerialFifo`]: this field stays `None` (preserves
    ///   pre-ADR-040 byte-equivalence — H95 pin).
    ///
    /// **Iter-C2c-cont (Path A) scope**: the parallel provisioning
    /// surface is the COMPLETE scope of this iter — the per-request
    /// forward path (`forward_prefill_with_soft_tokens_slot_aware`'s
    /// `INVESTIGATION_ENV.hybrid_kv` dispatch-fork branch at line 2562)
    /// still surfaces typed `MultiSeqError::CapabilityUnsupported` named
    /// `iter-B4c-kernel-iter-2B per ADR-040 §6.1.32`.
    /// iter-B4c-kernel-iter-2B will refactor that branch to consume
    /// THIS field via `MlxBuffer::slice_view` at the F16-K
    /// `slot_id.0 * nkv * cap * hd * 2`-byte offset (mirror of the
    /// HB-encoded iter-2A-cont scheme).
    pub multi_seq_kv_hybrid:
        Option<Vec<crate::inference::models::gemma4::kv_cache::MultiSeqHybridKvBuffers>>,
    /// ADR-040 iter-C2c-cont-cont (2026-05-30) — multi-seq dense F32 KV
    /// scaffold sibling.  Provisioned IFF `INVESTIGATION_ENV.use_dense`
    /// is true at SlotAware spawn time (HF2Q_USE_DENSE=1 opt-in
    /// pre-default surface).  Consumed by the iter-2D dispatch-fork
    /// branch in [`forward_prefill_with_soft_tokens_slot_aware`] +
    /// iter-2-decode-D dispatch-fork branch in [`forward_decode_slot_aware`]
    /// via `MlxBuffer::slice_view` slot-view mount on `self.dense_kvs`.
    ///
    /// `None` when HF2Q_USE_DENSE=0 (default) — the iter-2D dispatch-fork
    /// branch surfaces typed `iter-C2c-cont-cont-invariant-violated`
    /// defense-in-depth at this case (operator who flipped the env
    /// post-LazyLock-cache would land here).  See ADR-040 §6.1.46.
    pub multi_seq_kv_dense:
        Option<Vec<crate::inference::models::gemma4::kv_cache::MultiSeqDenseKvBuffers>>,
    /// ADR-040 iter-C2c-cont-cont (2026-05-30) — multi-seq legacy 4-bit
    /// nibble-packed KV scaffold sibling.  Provisioned IFF the
    /// HF2Q_TQ_CODEBOOK_BITS=4 env gate is engaged at SlotAware spawn
    /// time (opt-in pre-default since ADR-007 default-on TQ-8-bit
    /// correction 2026-04-24).  Consumed by the iter-2C dispatch-fork
    /// branch in [`forward_prefill_with_soft_tokens_slot_aware`] +
    /// iter-2-decode-D dispatch-fork branch in [`forward_decode_slot_aware`]
    /// via `MlxBuffer::slice_view` slot-view mount on `self.kv_caches`
    /// (Vec swap pattern via `std::mem::replace`).
    ///
    /// `None` when HF2Q_TQ_CODEBOOK_BITS != "4" (default) — the iter-2C
    /// dispatch-fork branch surfaces typed
    /// `iter-C2c-cont-cont-invariant-violated` defense-in-depth at this
    /// case.  See ADR-040 §6.1.46.
    pub multi_seq_kv_mlx:
        Option<Vec<crate::inference::models::gemma4::kv_cache::MultiSeqMlxKvCache>>,
}

impl LoadedModel {
    pub fn model_id(&self) -> &str {
        match self {
            LoadedModel::Gemma(g) => &g.model_id,
            LoadedModel::Qwen35(q) => &q.model_id,
            LoadedModel::Qwen3VlText(v) => &v.model_id,
            LoadedModel::Deepseek4(d) => &d.model_id,
        }
    }
    pub fn context_length(&self) -> Option<usize> {
        match self {
            LoadedModel::Gemma(g) => g.context_length,
            LoadedModel::Qwen35(q) => q.context_length,
            LoadedModel::Qwen3VlText(v) => v.context_length,
            LoadedModel::Deepseek4(d) => d.context_length,
        }
    }
    pub fn quant_type(&self) -> Option<&str> {
        match self {
            LoadedModel::Gemma(g) => g.quant_type.as_deref(),
            LoadedModel::Qwen35(q) => q.quant_type.as_deref(),
            LoadedModel::Qwen3VlText(v) => v.quant_type.as_deref(),
            LoadedModel::Deepseek4(d) => d.quant_type.as_deref(),
        }
    }
    pub fn model_path(&self) -> &Path {
        match self {
            LoadedModel::Gemma(g) => &g.model_path,
            LoadedModel::Qwen35(q) => &q.model_path,
            LoadedModel::Qwen3VlText(v) => &v.model_path,
            LoadedModel::Deepseek4(d) => &d.model_path,
        }
    }
    pub fn hidden_size(&self) -> usize {
        match self {
            LoadedModel::Gemma(g) => g.weights.hidden_size,
            LoadedModel::Qwen35(q) => q.hidden_size,
            LoadedModel::Qwen3VlText(v) => v.hidden_size,
            LoadedModel::Deepseek4(d) => d.model.cfg.hidden_size as usize,
        }
    }
    pub fn vocab_size(&self) -> usize {
        match self {
            LoadedModel::Gemma(g) => g.weights.vocab_size,
            LoadedModel::Qwen35(q) => q.vocab_size,
            LoadedModel::Qwen3VlText(v) => v.vocab_size,
            LoadedModel::Deepseek4(d) => d.model.cfg.vocab_size as usize,
        }
    }
    pub fn tokenizer(&self) -> &Tokenizer {
        match self {
            LoadedModel::Gemma(g) => &g.tokenizer,
            LoadedModel::Qwen35(q) => &q.tokenizer,
            LoadedModel::Qwen3VlText(v) => &v.tokenizer,
            LoadedModel::Deepseek4(d) => &d.tokenizer,
        }
    }
    pub fn chat_template(&self) -> &str {
        match self {
            LoadedModel::Gemma(g) => &g.chat_template,
            LoadedModel::Qwen35(q) => &q.chat_template,
            LoadedModel::Qwen3VlText(v) => &v.chat_template,
            LoadedModel::Deepseek4(d) => &d.chat_template,
        }
    }

    /// ADR-017 §F4 — GGUF provenance for the loaded model.  Both
    /// variants capture provenance at GGUF-open time via
    /// `crate::core::provenance::detect(&gguf)`.  Gemma consumes it for
    /// dense KV-spill namespacing today; Qwen35 stores the same fact even
    /// though its hybrid KV-spill descriptor is a later ADR-017 phase.
    pub fn provenance(&self) -> crate::core::provenance::Provenance {
        match self {
            LoadedModel::Gemma(g) => g.provenance.clone(),
            LoadedModel::Qwen35(q) => q.provenance.clone(),
            LoadedModel::Qwen3VlText(v) => v.provenance.clone(),
            LoadedModel::Deepseek4(d) => d.provenance.clone(),
        }
    }
    pub fn eos_token_ids(&self) -> &[u32] {
        match self {
            LoadedModel::Gemma(g) => &g.eos_token_ids,
            LoadedModel::Qwen35(q) => &q.eos_token_ids,
            LoadedModel::Qwen3VlText(v) => &v.eos_token_ids,
            LoadedModel::Deepseek4(d) => &d.eos_token_ids,
        }
    }
    pub fn load_duration(&self) -> Duration {
        match self {
            LoadedModel::Gemma(g) => g.load_duration,
            LoadedModel::Qwen35(q) => q.load_duration,
            LoadedModel::Qwen3VlText(v) => v.load_duration,
            LoadedModel::Deepseek4(d) => d.load_duration,
        }
    }
    /// Prompt cache is Gemma-only in iter-215 MVP.  The Qwen35 worker
    /// arm returns 501 before any prompt-cache logic runs, so the
    /// `None` returned here is unreachable on the Qwen35 path.  Wedge-3
    /// (full Qwen3.5/3.6 inference) will revisit caching scope.
    /// iter-228a Qwen3VlText: same shape — sentinel-route returns 501
    /// before prompt cache could fire.
    pub fn prompt_cache(&self) -> Option<&PromptCache> {
        match self {
            LoadedModel::Gemma(g) => Some(&g.prompt_cache),
            LoadedModel::Qwen35(_) => None,
            LoadedModel::Qwen3VlText(_) => None,
            LoadedModel::Deepseek4(_) => None,
        }
    }
}

/// Generation-affecting parameters that must all match for a cache hit.
///
/// Wave-2.5 B5 (HIGH-7): the iter-96 cache keyed only on prompt tokens,
/// silently ignoring `max_tokens`, `stop_strings`, `logit_bias`, and
/// `grammar`.  Two requests with the same prompt but different max_tokens
/// would incorrectly replay a shorter (or longer) cached response.  This
/// newtype makes all generation-affecting fields part of the equality
/// check.
///
/// Wave-2.6 W-ε (B5 honest closure): wave-2.5 commit overstated B5
/// closure.  The key still excluded `frequency_penalty`, `presence_penalty`,
/// `min_p`, `grammar_kind`, `tool_call_policy`, `logprobs`,
/// `top_logprobs`, and `parallel_tool_calls`.  Option A (mantra): every
/// generation-affecting parameter is included in the key — even parameters
/// not yet wired into the sampler — so that future wiring never introduces
/// a silent stale-replay bug.
///
/// Inventory of ALL `SamplingParams` fields and their cache treatment:
///
/// **Excluded (bypass gate already handles these):**
/// - `temperature` — non-zero bypasses cache; never reaches key check
/// - `top_p` — < 1.0 bypasses cache
/// - `top_k` — > 0 bypasses cache
/// - `repetition_penalty` — ≠ 1.0 bypasses cache
/// - `seed` — Some(_) bypasses cache
/// - `token_bytes` — derived from `grammar`; identical iff `grammar` is identical
///
/// **Included (affect model output or response shape):**
/// - `max_tokens` — early-stop trigger
/// - `stop_strings` — early-stop trigger
/// - `logit_bias` — additive shift applied before argmax (wired)
/// - `grammar` — token-validity mask (wired)
/// - `grammar_kind` — ResponseFormat vs ToolCallBody changes enforcement
///   timing; wired in wave-2.6 W-α5 (same grammar, different kind →
///   completely different output for tool-call vs. unconditional paths)
/// - `frequency_penalty` — penalty applied to sampler (plumbed, not yet wired
///   into greedy path; included now so future wiring is safe)
/// - `presence_penalty` — same as frequency_penalty
/// - `min_p` — min-p sampling cutoff (plumbed, not yet wired; included for
///   forward-compatibility)
/// - `tool_call_policy` — Auto vs Constrained changes error-promotion on
///   parse failure; a cached Auto replay served to a Constrained caller
///   would silently suppress error signalling
/// - `logprobs` — changes response shape (logprob data in choices)
/// - `top_logprobs` — changes response shape (number of top alternatives)
/// - `parallel_tool_calls` — plumbed, not yet wired; included for
///   forward-compatibility
#[derive(Debug, Clone, PartialEq)]
pub struct PromptCacheKey {
    pub max_tokens: usize,
    pub stop_strings: Vec<String>,
    /// Sorted key-value pairs from `logit_bias` so that two maps with
    /// identical contents compare equal regardless of insertion order.
    /// The bias values are stored as `f32` bit-patterns (via
    /// `to_bits()`) to enable structural equality without floating-point
    /// surprises.  Finite `f32` bias values are the only ones with
    /// meaningful semantics; `f32::NAN` keys would be a caller bug.
    pub logit_bias_sorted: Vec<(u32, u32)>,
    /// Structural equality: `GretElement: PartialEq` + `Grammar: PartialEq`.
    /// `None` means no grammar constraint; `Some(g)` means the entire
    /// GBNF rule set must match.
    pub grammar: Option<super::grammar::Grammar>,
    /// ResponseFormat vs ToolCallBody — same Grammar but different kind
    /// produces different enforcement timing and therefore different output.
    /// Wired in wave-2.6 W-α5.
    pub grammar_kind: GrammarKind,
    /// Stored as bit-pattern to allow structural equality without f32 surprises.
    /// Default 0.0 → 0u32. Plumbed but not yet wired into greedy path;
    /// included now for forward-safe wiring.
    pub frequency_penalty_bits: u32,
    /// Same treatment as `frequency_penalty_bits`.
    pub presence_penalty_bits: u32,
    /// Min-p sampling cutoff bit-pattern. 0.0 → 0u32.
    pub min_p_bits: u32,
    /// Auto vs Constrained — affects error-promotion on parse failure.
    pub tool_call_policy: ToolCallPolicy,
    /// `true` = include per-token logprob data in response. Changes response
    /// shape; different callers expect different response structures.
    pub logprobs: bool,
    /// Number of top-alternatives to report per token. Changes response shape.
    pub top_logprobs: u32,
    /// Multi-tool-call flag. Plumbed, not yet wired; included for
    /// forward-safe wiring.
    pub parallel_tool_calls: bool,
}

impl PromptCacheKey {
    /// Construct from a `SamplingParams`.
    pub fn from_params(params: &SamplingParams) -> Self {
        let mut bias_sorted: Vec<(u32, u32)> = params
            .logit_bias
            .iter()
            .map(|(&tok, &bias)| (tok, bias.to_bits()))
            .collect();
        bias_sorted.sort_unstable_by_key(|&(tok, _)| tok);
        Self {
            max_tokens: params.max_tokens,
            stop_strings: params.stop_strings.clone(),
            logit_bias_sorted: bias_sorted,
            grammar: params.grammar.clone(),
            grammar_kind: params.grammar_kind,
            frequency_penalty_bits: params.frequency_penalty.to_bits(),
            presence_penalty_bits: params.presence_penalty.to_bits(),
            min_p_bits: params.min_p.to_bits(),
            tool_call_policy: params.tool_call_policy,
            logprobs: params.logprobs,
            top_logprobs: params.top_logprobs,
            parallel_tool_calls: params.parallel_tool_calls,
        }
    }
}

/// One captured streaming-emit event, for fragment-replay (W-A2.1).
///
/// The `replay_cached_streaming_response` helper today re-runs the
/// ReasoningSplitter + ToolCallSplitter pipeline over `PromptCache::text`
/// to reconstruct the SSE event sequence on a cache hit.  That preserves
/// **structural shape** (Content / Reasoning / ToolCallDelta) but loses
/// **per-token boundaries**: a 100-token live decode emits ~100 deltas;
/// the replay emits one big splitter pass.
///
/// Fragment-replay (W-A2.1–W-A2.4) closes that gap by capturing each
/// `GenerationEvent::Delta` / `::ToolCallDelta` actually emitted during the
/// streaming decode into a `Vec<CachedFragment>` stored alongside `text`.
/// On a hit, the replay emits each `CachedFragment` directly as the matching
/// `GenerationEvent`, byte-identical to the live event stream.
///
/// # Why not `Vec<(DeltaKind, String)>`?
///
/// `DeltaKind` is `Content | Reasoning` only (sse.rs:50).  Tool-call deltas
/// carry richer structural fields (`index`, `id`, `call_type`, `name`,
/// `arguments`) that the deferral note's `(DeltaKind, String)` shape cannot
/// represent.  This enum is a 1:1 mirror of the `GenerationEvent` variants
/// the streaming path emits — lossless capture + lossless replay.
///
/// # Why no `ToolCallClose`?
///
/// The live path does not emit a "close" event — `tc_index += 1` and
/// `saw_tool_call = true` are bookkeeping that flips the terminal `Done`
/// event's `finish_reason` to `"tool_calls"`.  The cached `finish_reason`
/// already lives in `PromptCache::finish_reason` and is replayed on the
/// `Done` chunk, so no separate Close fragment is needed.
#[derive(Debug, Clone, PartialEq)]
pub enum CachedFragment {
    /// `GenerationEvent::Delta { kind: DeltaKind::Content, text }`.
    Content(String),
    /// `GenerationEvent::Delta { kind: DeltaKind::Reasoning, text }`.
    Reasoning(String),
    /// `GenerationEvent::ToolCallDelta { index, id, call_type, name, arguments }`.
    /// All five fields preserved verbatim — first-chunk shape (id+name+
    /// call_type, arguments=None) and subsequent args-chunk shape
    /// (everything None except arguments=Some(fragment)) both round-trip
    /// without loss.
    ToolCallDelta {
        index: usize,
        id: Option<String>,
        call_type: Option<String>,
        name: Option<String>,
        arguments: Option<String>,
    },
}

/// W-A2.2 streaming-emit sink with optional fragment capture.
///
/// Wraps the `tokio::sync::mpsc::Sender<GenerationEvent>` that streaming
/// helpers (`generate_stream_once`, `route_content`, `emit_fragment`,
/// `finalize_streaming_tool_state`, `ToolCallStreamEmitter::*`,
/// `emit_streaming_tool_call_close`, `replay_cached_streaming_response`)
/// previously took as `&mpsc::Sender<...>`.
///
/// **Why a wrapper:** the W-A2.2 capture must mirror EVERY emitted
/// `GenerationEvent::Delta` / `::ToolCallDelta` into a parallel
/// `Vec<CachedFragment>` accumulator without missing a callsite.  Threading
/// `Option<&RefCell<Vec<CachedFragment>>>` as a separate parameter through
/// every helper (the alternative considered) costs the same number of
/// signature changes AND adds a manual `if let Some(cap) = ...` block at
/// every send site.  Centralising both forwarding and capture inside
/// `EventSink::blocking_send` makes "every emit gets captured" structural
/// rather than convention-bound.
///
/// **Capture ordering:** the capture push happens BEFORE the channel send
/// so a client-disconnect mid-decode (channel send returns `Err`) does NOT
/// drop the fragment — the cache write at end-of-stream still records the
/// full prefix the cache will later replay.  This matches the
/// `accumulated_text.push_str` ordering at engine.rs:5161,5260 (text is
/// pushed before `emit_fragment`).
///
/// **`call_type` normalisation:** none — the field is preserved verbatim
/// (`Option<String>`).  Converting back to `GenerationEvent` is lossless.
///
/// **`Done` / `Error` / `Logprobs` are NOT captured:** they are
/// terminal-shape control events whose state is reconstructed from
/// `cached.{finish_reason, prompt_tokens, completion_tokens, ...}` at
/// replay time.  Capturing them would double-emit on hit.
pub(super) struct EventSink<'a> {
    sender: &'a tokio::sync::mpsc::Sender<super::sse::GenerationEvent>,
    /// `Some(_)` — populated capture; mirror every Delta/ToolCallDelta.
    /// `None` — passive forwarder; events flow through unchanged.
    capture: Option<&'a std::cell::RefCell<Vec<CachedFragment>>>,
}

impl<'a> EventSink<'a> {
    /// Passive sink — no capture.  Used by `replay_cached_streaming_response`
    /// (replay does not capture; it re-emits already-captured fragments) and
    /// by tests / qwen35 callsites that don't participate in the Gemma
    /// streaming-origin store path.
    pub(super) fn new(sender: &'a tokio::sync::mpsc::Sender<super::sse::GenerationEvent>) -> Self {
        Self {
            sender,
            capture: None,
        }
    }

    /// Capture sink — every `Delta` / `ToolCallDelta` is mirrored into
    /// `capture` before forwarding to `sender`.  Used by
    /// `generate_stream_once` for the streaming-origin store path.
    pub(super) fn with_capture(
        sender: &'a tokio::sync::mpsc::Sender<super::sse::GenerationEvent>,
        capture: &'a std::cell::RefCell<Vec<CachedFragment>>,
    ) -> Self {
        Self {
            sender,
            capture: Some(capture),
        }
    }

    /// Forward an event to the underlying channel, mirroring into the
    /// capture vec first if active.  Same signature shape as
    /// `tokio::sync::mpsc::Sender::blocking_send` so helper bodies that
    /// previously called `events.blocking_send(...)` need NO body edit.
    pub(super) fn blocking_send(
        &self,
        ev: super::sse::GenerationEvent,
    ) -> Result<(), tokio::sync::mpsc::error::SendError<super::sse::GenerationEvent>> {
        // Mirror BEFORE forwarding — see struct doc for ordering rationale.
        if let Some(cap) = self.capture {
            match &ev {
                super::sse::GenerationEvent::Delta {
                    kind: super::sse::DeltaKind::Content,
                    text,
                } => cap.borrow_mut().push(CachedFragment::Content(text.clone())),
                super::sse::GenerationEvent::Delta {
                    kind: super::sse::DeltaKind::Reasoning,
                    text,
                } => cap
                    .borrow_mut()
                    .push(CachedFragment::Reasoning(text.clone())),
                super::sse::GenerationEvent::ToolCallDelta {
                    index,
                    id,
                    call_type,
                    name,
                    arguments,
                } => cap.borrow_mut().push(CachedFragment::ToolCallDelta {
                    index: *index,
                    id: id.clone(),
                    call_type: call_type.clone(),
                    name: name.clone(),
                    arguments: arguments.clone(),
                }),
                // Done / Error / Logprobs: NOT captured.  See struct doc.
                _ => {}
            }
        }
        self.sender.blocking_send(ev)
    }
}

/// Single-slot prompt cache (Phase 2a Task #7, iter-96).
///
/// **Iter-96 scope: full-equality + temperature=0 cache.**  When the
/// next chat request's prompt_tokens exactly matches `tokens` AND the
/// caller's `temperature == 0` (deterministic decode), the cache
/// short-circuits the entire prefill+decode and replays the previous
/// response.  Useful for retries (network failures), eval consistency,
/// repeated benchmarks, idempotent agentic loops.
///
/// **Iter-97+ scope: LCP-based partial-prefill resume.**  Compute the
/// longest common prefix between the new prompt and `tokens`, set
/// `kv_caches[*].write_pos = LCP`, pre-warm `dense_kvs[0..LCP)` by
/// dequantizing `kv_caches[0..LCP)` via `tq_dequantize_kv`, then run
/// `forward_prefill` for tokens `[LCP..N)`.  Reports `cached_tokens =
/// LCP` (any value `0 ≤ LCP ≤ prompt_tokens`).  Defers to a later
/// iteration because the dequant pre-warm is non-trivial — the iter-96
/// full-equality cache is a real, shippable subset.
///
/// Sampling (`temperature > 0`) **bypasses the cache** even on full
/// equality — replaying the deterministic-greedy decoded text under a
/// sampling request would silently violate the user's expectation of
/// per-call variation.  No cache write happens on sampling-mode hits
/// either; sampling completions are always re-generated.
///
/// Grammar-constrained requests (`response_format=json_object` /
/// `json_schema`) follow the same rule: greedy + matching prompt =
/// cache hit; sampling = cache bypass.  The grammar runtime state
/// at the end of generation is NOT cached (would over-constrain a
/// future hit if the cached grammar differed from the new request's).
///
/// # Design invariants (DO NOT re-create a separate prompt_cache module)
///
/// A simpler `PromptCache` with only `lcp_len` / `update` / `clear` methods
/// was prototyped in `src/serve/api/prompt_cache.rs` (ADR-005 Task #7 first
/// cut) and deleted in wave-1.5 (2026-04-26) because:
///
/// 1. **Full-equality is the shipped contract.**  The iter-96 cache fires
///    only when `new_prompt == cached_prompt` exactly.  The prototype's
///    LCP algorithm is correct but belongs to the iter-97+ scope (LCP-based
///    partial-prefill resume) which needs a `forward_decode` refactor to
///    expose the KV write position — that refactor is deferred.
///
/// 2. **Full-response-replay, not partial-skip.**  On a cache hit the
///    worker returns the complete cached `GenerationResult` (`text`,
///    `reasoning_text`, `completion_tokens`, `finish_reason`, …) without
///    running the decoder at all.  `cached_tokens = prompt_len` surfaces
///    in the OpenAI usage shape per the spec.
///
/// 3. **Owned by the worker thread.**  `PromptCache` lives inside
///    `LoadedModel` (field `prompt_cache`), which is exclusive to the
///    single worker thread; no synchronization needed.  Moving it to a
///    shared module would require Arc/Mutex overhead without benefit.
///
/// Future LCP-based work belongs here, extending `lookup`/`store`.
#[derive(Debug, Clone)]
pub struct PromptCache {
    /// The previous request's prompt token sequence (post-rendering,
    /// post-tokenization).  Empty on a fresh worker (no prior request).
    pub tokens: Vec<u32>,
    /// Wave-2.5 B5: all generation-affecting params from the previous
    /// request.  A new request must match both `tokens` AND `key` to
    /// get a cache hit.
    pub key: PromptCacheKey,
    /// The text the previous request emitted (post reasoning-marker
    /// split).  This is what gets replayed on a cache hit.
    pub text: String,
    /// The `reasoning_text` field from the previous result.  Replayed
    /// alongside `text` so the response shape matches the original.
    pub reasoning_text: Option<String>,
    /// Number of completion tokens the previous request emitted.
    pub completion_tokens: usize,
    /// Reasoning-token count from the previous result.
    pub reasoning_tokens: Option<usize>,
    /// `"stop"` | `"length"` from the previous result.
    pub finish_reason: &'static str,
    /// Captured per-emit `GenerationEvent` sequence from the previous
    /// streaming decode, for fragment-replay (W-A2.1–W-A2.4).
    ///
    /// `None` for a fresh cache, OR for any entry whose origin was
    /// non-streaming (`generate_once_with_soft_tokens`) — non-streaming
    /// has no per-token trace, so the splitter-rerun replay path is the
    /// honest minimum (Worker AA design §3b option (a)).
    ///
    /// `Some(frags)` for streaming-origin entries — the replay path
    /// emits each `CachedFragment` directly, byte-identical to the live
    /// event stream.  Single-slot cache ⇒ ~5–10 KB worst-case footprint
    /// (Worker AA design §3e).
    pub fragments: Option<Vec<CachedFragment>>,
}

impl Default for PromptCache {
    fn default() -> Self {
        Self::new()
    }
}

impl PromptCache {
    /// Empty cache — initial state for a fresh worker.
    pub fn new() -> Self {
        Self {
            tokens: Vec::new(),
            key: PromptCacheKey {
                max_tokens: 0,
                stop_strings: Vec::new(),
                logit_bias_sorted: Vec::new(),
                grammar: None,
                grammar_kind: GrammarKind::default(),
                frequency_penalty_bits: 0u32,
                presence_penalty_bits: 0u32,
                min_p_bits: 0u32,
                tool_call_policy: ToolCallPolicy::Auto,
                logprobs: false,
                top_logprobs: 0,
                parallel_tool_calls: true,
            },
            text: String::new(),
            reasoning_text: None,
            completion_tokens: 0,
            reasoning_tokens: None,
            finish_reason: "length",
            fragments: None,
        }
    }

    /// Cache check: returns the cached result if and only if
    /// `prompt_tokens` exactly equals the cached prompt, the caller
    /// is in greedy decode mode (temperature = 0, no sampling-only
    /// fields set), AND all generation-affecting params match the
    /// cached key.
    ///
    /// Wave-2.5 B5 / Wave-2.6 W-ε: `PromptCacheKey` now covers the
    /// complete inventory of generation-affecting params:
    /// `max_tokens`, `stop_strings`, `logit_bias`, `grammar`,
    /// `grammar_kind`, `frequency_penalty`, `presence_penalty`,
    /// `min_p`, `tool_call_policy`, `logprobs`, `top_logprobs`,
    /// `parallel_tool_calls`.  See `PromptCacheKey` doc for rationale.
    pub fn lookup(
        &self,
        prompt_tokens: &[u32],
        params: &SamplingParams,
    ) -> Option<GenerationResult> {
        self.lookup_with_fragments(prompt_tokens, params)
            .map(|(result, _frags)| result)
    }

    /// Streaming-aware cache check: returns the cached `GenerationResult`
    /// AND a borrow of the captured fragment vec (if any) so the caller
    /// can branch its replay strategy on origin (W-A2.3).
    ///
    /// Same eligibility gate as `lookup`.  Streaming-origin entries
    /// (stored via `store_with_fragments(.., Some(_))`) yield
    /// `Some((result, Some(frags)))` — replay emits each
    /// `CachedFragment` directly as the matching `GenerationEvent`,
    /// byte-identical to the live event stream (W-A2.3 fragments
    /// branch).  Non-streaming origin entries (stored via plain
    /// `store(...)`) yield `Some((result, None))` — replay falls
    /// through to the splitter-rerun branch, preserving the Wave-3.5
    /// HIGH-2 splitter `tail_buf` drain (engine.rs:4332).
    ///
    /// The fragment vec is borrowed (`&Vec<CachedFragment>`) rather
    /// than cloned to keep the cache-hit fast path zero-copy.  Replay
    /// streams each fragment as a fresh `GenerationEvent` whose String
    /// payloads are cloned at emit time only.
    pub fn lookup_with_fragments(
        &self,
        prompt_tokens: &[u32],
        params: &SamplingParams,
    ) -> Option<(GenerationResult, Option<&Vec<CachedFragment>>)> {
        // Bypass for any non-greedy mode.  These all introduce per-call
        // variance that a cached replay would silently erase.
        if params.temperature > 0.0
            || params.top_k > 0
            || params.top_p < 1.0
            || params.repetition_penalty != 1.0
            || params.seed.is_some()
        {
            return None;
        }
        if self.tokens.is_empty() || self.tokens.as_slice() != prompt_tokens {
            return None;
        }
        // Wave-2.5 B5: generation-affecting params must also match.
        let request_key = PromptCacheKey::from_params(params);
        if self.key != request_key {
            return None;
        }
        let result = GenerationResult {
            text: self.text.clone(),
            reasoning_text: self.reasoning_text.clone(),
            prompt_tokens: prompt_tokens.len(),
            completion_tokens: self.completion_tokens,
            reasoning_tokens: self.reasoning_tokens,
            finish_reason: self.finish_reason,
            // Cache hit: prefill and decode were both skipped — report
            // zero wall-clock for both phases.  TTFT effectively becomes
            // the cache lookup time (~1µs), surfaced as 0 in the response.
            prefill_duration: Duration::ZERO,
            decode_duration: Duration::ZERO,
            cached_tokens: prompt_tokens.len(),
            logprobs: None,
        };
        Some((result, self.fragments.as_ref()))
    }

    /// Cache write: store this request's result so the next
    /// equal-prompt + greedy + equal-params request can short-circuit.
    ///
    /// Same eligibility gate as `lookup` — sampling-mode requests are
    /// not cached (storing them would mean a future greedy request
    /// could replay a sampling outcome, violating determinism).
    ///
    /// **Fragment-replay (W-A2.1)**: this entry-point sets
    /// `fragments = None`, which is the correct behaviour for
    /// non-streaming origin (`generate_once_with_soft_tokens`) — no
    /// per-token trace exists.  The streaming origin uses
    /// `store_with_fragments` to capture the per-emit
    /// `CachedFragment` sequence.
    pub fn store(
        &mut self,
        prompt_tokens: &[u32],
        params: &SamplingParams,
        result: &GenerationResult,
    ) {
        self.store_with_fragments(prompt_tokens, params, result, None);
    }

    /// Cache write with optional captured fragment sequence (W-A2.1).
    ///
    /// `fragments == Some(frags)` records the per-emit `GenerationEvent`
    /// trace for fragment-replay (W-A2.3) — the replay path emits each
    /// `CachedFragment` directly as the matching `GenerationEvent`,
    /// byte-identical to the live event stream.
    ///
    /// `fragments == None` records the legacy text-only entry whose
    /// replay re-runs the splitter pipeline over `text` (preserves
    /// structural shape, loses token boundaries).  This is the honest
    /// minimum for non-streaming origin, where no per-token trace exists
    /// (Worker AA design §3b option (a)).
    ///
    /// Same eligibility gate as `lookup` and `store` — sampling-mode
    /// requests are NOT cached, including those carrying captured
    /// fragments (storing them would mean a future greedy request could
    /// replay a sampling outcome, violating determinism).
    pub fn store_with_fragments(
        &mut self,
        prompt_tokens: &[u32],
        params: &SamplingParams,
        result: &GenerationResult,
        fragments: Option<Vec<CachedFragment>>,
    ) {
        if params.temperature > 0.0
            || params.top_k > 0
            || params.top_p < 1.0
            || params.repetition_penalty != 1.0
            || params.seed.is_some()
        {
            return;
        }
        self.tokens = prompt_tokens.to_vec();
        self.key = PromptCacheKey::from_params(params);
        self.text = result.text.clone();
        self.reasoning_text = result.reasoning_text.clone();
        self.completion_tokens = result.completion_tokens;
        self.reasoning_tokens = result.reasoning_tokens;
        self.finish_reason = result.finish_reason;
        self.fragments = fragments;
    }
}

/// Options for `LoadedModel::load`. Mirrors `cli::ServeArgs` without pulling
/// the CLI type into this module.
#[derive(Debug, Clone)]
pub struct LoadOptions {
    pub model_path: PathBuf,
    pub tokenizer_path: Option<PathBuf>,
    pub config_path: Option<PathBuf>,
    /// ADR-020 AC#5 Iter D — optional path to a DWQ-trained mlx-affine
    /// safetensors file.  When `Some`, applied as an overlay over the
    /// GGUF-loaded weights via `MlxModelWeights::apply_dwq_overlay`,
    /// replacing each trained Linear with the DWQ output.
    ///
    /// Only the dense families (Gemma 4) consume this in Iter D; the
    /// qwen35moe path will gain DWQ-overlay support in Iter C2.
    pub dwq_overlay_path: Option<PathBuf>,

    /// ADR-027 Phase A iter-6b.2 — root directory for cold-process LCP
    /// resume on the qwen35 family. When `Some`, `Qwen35LoadedModel::load`
    /// constructs a `Qwen35DiskPersistor` rooted here; the lcp_registry
    /// store call sites write through to disk on insert (per-cfg
    /// fingerprint subdirs); the first prefill of any new cfg lazily
    /// hydrates pre-existing snapshots into the in-memory registry.
    /// Sourced from `HF2Q_KV_PERSIST` env at cmd_serve / cmd_generate
    /// startup; `None` keeps the legacy in-process-only behavior.
    pub kv_persist_dir: Option<PathBuf>,
}

impl LoadedModel {
    /// Dispatcher: open the GGUF header, read `general.architecture`,
    /// route to the matching variant's `load` constructor.
    ///
    /// Iter-215 Wedge-2: replaces the flat-struct constructor.  The
    /// pre-iter-215 body is now `GemmaLoadedModel::load`; the new
    /// `Qwen35LoadedModel::load` lives in `engine_qwen35.rs`.  The
    /// SERVE-side `load_engine` (`src/serve/mod.rs`) used to bail
    /// before this constructor for qwen35 / qwen35moe arches; iter-215
    /// replaces that wedge with actual dispatch.
    pub fn load(opts: &LoadOptions) -> Result<Self> {
        let model_path = &opts.model_path;
        anyhow::ensure!(
            model_path.exists(),
            "Model not found: {}",
            model_path.display()
        );
        // Header-only parse — cheap; no tensor read.
        let gguf = mlx_native::gguf::GgufFile::open(model_path)
            .map_err(|e| anyhow::anyhow!("GGUF open: {e}"))?;
        let arch = gguf
            .metadata_string("general.architecture")
            .map(|s| s.to_string())
            .unwrap_or_default();
        // Wedge-4 / iter-227 (2026-05-02): originally a runtime
        // actionable-error dispatch shim that bailed on Qwen3-VL
        // arches because the LM forward path wasn't wired yet.
        //
        // **iter-228a (2026-05-02)** replaces the bail with a real
        // dispatch arm: dense Qwen3-VL GGUFs now route through
        // `Qwen3VlTextLoadedModel::load` (load surface lands; chat
        // arm continues to short-circuit to 501 via the
        // `QWEN3VL_TEXT_FORWARD_PENDING_SENTINEL` until iter-228b
        // wires the actual transformer forward).
        //
        // MoE Qwen3-VL still bails at this site (no convert pipeline
        // emits it; the dense-only loader cannot consume an MoE GGUF
        // structurally), with the same operator-actionable message.
        use crate::inference::models::qwen35::{is_qwen3_vl_arch, is_qwen3_vl_moe_arch};
        if is_qwen3_vl_arch(arch.as_str()) {
            if is_qwen3_vl_moe_arch(arch.as_str()) {
                anyhow::bail!(
                    "Qwen3-VL (MoE, general.architecture = {arch:?}) GGUFs are recognized \
                     but no convert pipeline currently emits this variant; the dense Qwen3-VL \
                     LM loader is iter-228a scope and cannot consume an MoE GGUF structurally. \
                     For dense Qwen3-VL today, use a `qwen3_vl` / `qwen3vl` GGUF (e.g. \
                     `Qwen/Qwen3-VL-2B-Instruct`). Model: {}",
                    model_path.display(),
                );
            }
            // Dense Qwen3-VL — route through iter-228a's load path.
            let v = super::engine_qwen3vl::Qwen3VlTextLoadedModel::load(opts)?;
            return Ok(LoadedModel::Qwen3VlText(v));
        }
        match arch.as_str() {
            "qwen35" | "qwen35moe" => {
                let q = super::engine_qwen35::Qwen35LoadedModel::load(opts)?;
                Ok(LoadedModel::Qwen35(q))
            }
            "gemma4" => {
                let g = GemmaLoadedModel::load(opts)?;
                Ok(LoadedModel::Gemma(g))
            }
            "deepseek4" => {
                let d = super::engine_deepseek4::Deepseek4LoadedModel::load(opts)?;
                Ok(LoadedModel::Deepseek4(d))
            }
            "" => anyhow::bail!(
                "GGUF is missing required `general.architecture`; refusing to guess Gemma. \
                 Model: {}",
                model_path.display()
            ),
            other => anyhow::bail!(
                "unsupported GGUF general.architecture={other:?}; supported runtimes in this \
                 build are gemma4, qwen35, qwen35moe, dense qwen3_vl, and deepseek4. Model: {}",
                model_path.display()
            ),
        }
    }
}

impl GemmaLoadedModel {
    /// Perform the full Gemma 4 model-load pipeline: open GGUF, load weights
    /// into mlx-native, load the tokenizer, resolve the chat template, read
    /// the context length from metadata.
    ///
    /// This mirrors `cmd_generate`'s load sequence (`src/serve/mod.rs:188-252`)
    /// so the two entrypoints are guaranteed to produce the same model state.
    /// Any future change to the load path belongs in a shared helper rather
    /// than duplicated here — maintainers: if you touch one, touch both.
    pub fn load(opts: &LoadOptions) -> Result<Self> {
        let load_start = Instant::now();
        // ADR-028 iter-461: opt-in sub-phase timing via HF2Q_LOAD_TIMING=1.
        // Surfaces where the 2.91 sec load_wall_clock is spent (per iter-460).
        let load_timing = std::env::var("HF2Q_LOAD_TIMING").as_deref() == Ok("1");
        let mut t_phase = Instant::now();

        let model_path = &opts.model_path;
        anyhow::ensure!(
            model_path.exists(),
            "Model not found: {}",
            model_path.display()
        );

        // ADR-022 P1.8 — GGUF is the single source of truth on the
        // inference path. The opts.config_path field is no longer
        // consulted here: the GGUF carries every Gemma4Config field
        // (verified key-by-key in `Gemma4Config::from_gguf`). The legacy
        // config.json path lives on in the calibration / parity /
        // safetensors pipelines (parity_quality.rs:442, mod.rs:4351 +
        // 4509) which have no GGUF input.
        //
        // ADR-022 P1.11 — same single-source-of-truth principle for the
        // tokenizer: if the operator passes `--tokenizer <path>` or there's
        // a `tokenizer.json` next to the .gguf we honor it (legacy HF-checkout
        // ergonomics); otherwise we build directly from
        // `tokenizer.ggml.{tokens,merges,token_type,...}` GGUF metadata via
        // `gemma4::tokenizer::build_tokenizer_from_gguf`. Parity verified
        // by `tests/adr_022_phase1_p11_gemma4_tokenizer_parity.rs` (5/5
        // cases byte-identical to the on-disk tokenizer.json on the
        // abliterated Gemma4-A4B file).
        let tokenizer_path_opt =
            resolve_tokenizer_path_optional(model_path, opts.tokenizer_path.as_deref());

        // Open GGUF (header + metadata only).
        let gguf = mlx_native::gguf::GgufFile::open(model_path)
            .map_err(|e| anyhow::anyhow!("GGUF open: {e}"))?;
        if load_timing {
            tracing::info!(
                "[LOAD_TIMING] gguf_open={:.0}ms",
                t_phase.elapsed().as_secs_f64() * 1000.0
            );
            t_phase = Instant::now();
        }

        let config = Gemma4Config::from_gguf(&gguf)
            .context("Failed to derive Gemma4Config from GGUF metadata")?;
        if load_timing {
            tracing::info!(
                "[LOAD_TIMING] config_parse={:.0}ms",
                t_phase.elapsed().as_secs_f64() * 1000.0
            );
            t_phase = Instant::now();
        }

        // ADR-017 §F4 + ADR-005 iter-211: detect hf2q-origin provenance
        // (producer_version + source_sha256 + optional mmproj_sha256) at
        // GGUF-open time. Threaded into `Engine::spawn`'s
        // `KvSpillDescriptor` so the per-family spill factory can build
        // a strict `ModelFingerprint`. External GGUFs (no provenance
        // keys) yield `Provenance::External` and the spiller falls back
        // to the legacy `(repo, quant)` namespace.
        let provenance = crate::core::provenance::detect(&gguf);

        // Extract model id: prefer general.name, fall back to file stem.
        let model_id = gguf
            .metadata_string("general.name")
            .map(|s| s.to_string())
            .unwrap_or_else(|| {
                model_path
                    .file_stem()
                    .map(|s| s.to_string_lossy().into_owned())
                    .unwrap_or_else(|| "unknown".to_string())
            });

        // Context length: arch-prefixed metadata key.
        let arch = gguf.metadata_string("general.architecture").unwrap_or("");
        let context_length = if arch.is_empty() {
            None
        } else {
            gguf.metadata_u32(&format!("{arch}.context_length"))
                .map(|v| v as usize)
        };

        // Quant label: dominant non-fp tensor type. Promoted to
        // `crate::serve::load_info::infer_quant_label` per ADR-018 C1
        // (the prior inline body was byte-identical to the relocated
        // helper; behaviour is unchanged).
        let quant_type = crate::serve::load_info::infer_quant_label(&gguf);

        // Chat template: GGUF embedded or hardcoded fallback.
        let chat_template = gguf
            .metadata_string("tokenizer.chat_template")
            .map(|s| s.to_string())
            .unwrap_or_else(|| {
                // Phase A0.2b — API path uses minijinja rendering against
                // `messages` array; the CLI fallback template's literal
                // `{{PROMPT}}` placeholder is unbound under minijinja and
                // collapses every prompt to ~14 boilerplate tokens (root
                // cause of ADR-017 A0.2b flat-TTFT regression). The API
                // fallback iterates the messages array properly.
                tracing::warn!(
                    "Engine load: no GGUF `tokenizer.chat_template`; \
                     using API-path Gemma4 fallback (iterates messages \
                     array; supports multi-turn correctly)."
                );
                crate::serve::FALLBACK_GEMMA4_API_CHAT_TEMPLATE.to_string()
            });

        // Load GPU ctx + weights.
        //
        // ADR-018 C3: TTY-aware `LoadProgress::new(stderr_is_tty, verbosity, n_layers)`
        // replaces the previous hard-coded `LoadProgress::new(false, 1, n_layers)`
        // silent sentinel. The TTY-aware constructor is the same one
        // `cmd_generate` uses today (mod.rs:519-531). Behaviour:
        //
        //   - CLI default (`hf2q generate`, no -v): stderr is a TTY,
        //     verbosity=0 (tracing INFO not enabled by main.rs:124),
        //     progress reporter renders `\r loading i/n layers`.
        //   - CLI verbose (`hf2q generate -v`): stderr is a TTY,
        //     tracing::enabled!(INFO) is true → verbosity=1, reporter
        //     is silent (debug/info events from the loader cover detail).
        //   - SERVE default (`hf2q serve`): main.rs sets `hf2q=info`
        //     by default for serve mode (some configs); when stderr IS
        //     a TTY (interactive launch), tracing INFO is enabled →
        //     verbosity=1, reporter silent. When stderr ISN'T a TTY
        //     (systemd, docker), the reporter is silent regardless of
        //     verbosity. Either way: server output is clean.
        //   - Test contexts that capture stderr: stderr is rarely a
        //     TTY → reporter silent.
        if load_timing {
            tracing::info!(
                "[LOAD_TIMING] meta_misc(provenance+quant+template)={:.0}ms",
                t_phase.elapsed().as_secs_f64() * 1000.0
            );
            t_phase = Instant::now();
        }
        let mut ctx =
            GpuContext::new().map_err(|e| anyhow::anyhow!("mlx-native init failed: {e}"))?;
        if load_timing {
            tracing::info!(
                "[LOAD_TIMING] gpu_ctx_new={:.0}ms",
                t_phase.elapsed().as_secs_f64() * 1000.0
            );
            t_phase = Instant::now();
        }

        let n_layers = config.num_hidden_layers;
        let stderr_is_tty = std::io::IsTerminal::is_terminal(&std::io::stderr());
        let verbosity = if tracing::enabled!(tracing::Level::INFO) {
            1
        } else {
            0
        };
        let mut load_progress = header::LoadProgress::new(stderr_is_tty, verbosity, n_layers);
        let mut weights =
            MlxModelWeights::load_from_gguf(&gguf, &config, &mut ctx, &mut load_progress)?;
        if load_timing {
            tracing::info!(
                "[LOAD_TIMING] mlx_weights_load={:.0}ms",
                t_phase.elapsed().as_secs_f64() * 1000.0
            );
            t_phase = Instant::now();
        }

        // ADR-020 AC#5 Iter D — DWQ overlay (mlx-affine packed-U32
        // safetensors) applied after the GGUF load.  Replaces each
        // trained Linear's MlxQWeight with an affine-mode counterpart
        // that dispatches through `qmm_affine_t_packed_simd4_b4`.
        if let Some(overlay_path) = opts.dwq_overlay_path.as_ref() {
            let overridden = weights
                .apply_dwq_overlay(ctx.device(), overlay_path)
                .with_context(|| format!("DWQ overlay from {} failed", overlay_path.display()))?;
            tracing::info!(
                count = overridden,
                path = %overlay_path.display(),
                "DWQ overlay applied to GemmaLoadedModel"
            );
        }

        // Load tokenizer. ADR-022 P1.11: prefer on-disk tokenizer.json when
        // present (HF-checkout ergonomics), else build directly from GGUF
        // metadata. Both produce byte-identical token streams for Gemma4
        // per the parity test in
        // `tests/adr_022_phase1_p11_gemma4_tokenizer_parity.rs`.
        //
        // ADR-028 iter-466: opt-in `HF2Q_TOKENIZER_GGUF_EMBEDDED=1` forces
        // the GGUF-embedded path even when an on-disk tokenizer.json is
        // present.  Saves ~300 ms startup time per iter-461 (10% of total
        // load_wall_clock).
        //
        // ADR-028 iter-469 default-flipped: per iter-326 operator REFRAME #2
        // ("default should have the best things on") + 5 validation rounds:
        //   - iter-466: 200ms startup saving + coherence verified
        //   - iter-467: 10/10 varied prompts stable
        //   - iter-468: 5/5 byte-identical content on disk vs GGUF-embedded
        //   - tests/adr_022_phase1_p11_gemma4_tokenizer_parity.rs (5/5)
        //   - gemma4-only scope (gated inside Gemma load fn; no qwen35 risk)
        // Mirrors q6_K_NR2 (iter-326) + Phase 15 (iter-421) default-on pattern.
        // Opt out via `HF2Q_TOKENIZER_GGUF_EMBEDDED=0` / `=false` / `=off`.
        let force_gguf_tokenizer = std::env::var("HF2Q_TOKENIZER_GGUF_EMBEDDED")
            .ok()
            .map(|v| !matches!(v.to_ascii_lowercase().as_str(), "0" | "false" | "off"))
            .unwrap_or(true);
        let mut tokenizer = if force_gguf_tokenizer {
            crate::inference::models::gemma4::tokenizer::build_tokenizer_from_gguf(&gguf)
                .context("Failed to build Gemma4 tokenizer from GGUF metadata (HF2Q_TOKENIZER_GGUF_EMBEDDED=1)")?
        } else {
            match tokenizer_path_opt.as_ref() {
                Some(p) => Tokenizer::from_file(p).map_err(|e| {
                    anyhow::anyhow!("Failed to load tokenizer from {}: {e}", p.display())
                })?,
                None => {
                    crate::inference::models::gemma4::tokenizer::build_tokenizer_from_gguf(&gguf)
                        .context("Failed to build Gemma4 tokenizer from GGUF metadata")?
                }
            }
        };
        if load_timing {
            tracing::info!(
                "[LOAD_TIMING] tokenizer_init={:.0}ms",
                t_phase.elapsed().as_secs_f64() * 1000.0
            );
        }
        let _ = t_phase; // last phase consumes the timer; suppress unused warning
        let tokenizer_path = tokenizer_path_opt.unwrap_or_else(|| {
            // Synthetic sentinel for the load_info banner — communicates
            // "GGUF-embedded" in the path slot without misrepresenting an
            // on-disk file. Downstream code only uses this path for display.
            std::path::PathBuf::from("<gguf-embedded>")
        });
        tokenizer
            .with_truncation(None)
            .map_err(|e| anyhow::anyhow!("Failed to disable tokenizer truncation: {e}"))?;

        // EOS tokens: reuse the hardcoded list from cmd_generate. This is
        // what Gemma 4 uses; other models will be generalized alongside
        // per-model registration (Decision #21 — lands with tool calling).
        let eos_token_ids: Vec<u32> = vec![1, 106];

        let load_duration = load_start.elapsed();
        // ADR-018 C3: legacy `tracing::info!("Engine load: {} layers, ctx_len={:?}, load_time={:.1}s", ...)`
        // was deleted here. `emit_tracing(&info)` now surfaces the same
        // facts (`n_layers`, `max_context_length`, `load_wall_clock`) as
        // structured fields at every CLI/SERVE entry that constructs a
        // `LoadInfo`. The free-text format was incompatible with
        // `journalctl -u hf2q | jq` cross-arch filtering.

        Ok(Self {
            weights,
            ctx,
            config,
            model_id,
            model_path: model_path.clone(),
            tokenizer_path,
            context_length,
            quant_type,
            tokenizer,
            chat_template,
            eos_token_ids,
            load_duration,
            prompt_cache: PromptCache::new(),
            // ADR-017 Phase E.a — LCP registry. Capacity = 1 for v1.
            //
            // ADR-017 Phase E.a default-on — byte-budget LcpRegistry.
            //
            // Pre-iter-3 the payload was marker `()` (~0 bytes); iter-2
            // chose 16 entries for /cfa fan-out + last-16-turn
            // visibility at that cost. Iter-3 swapped the payload to
            // real `DenseKvBuffers` Arc clones — per-entry size on
            // Gemma 4 26B is ~4.8 GB (48 layers × ~100 MB/layer KV at
            // F32). Capacity 16 with that payload would budget ~77 GB
            // for LCP cache alone, OOM-class on a 128-GB M5 Max.
            //
            // This iter replaces entry-count cap with byte-budget
            // eviction. The budget is computed from `sysinfo`
            // `available_memory() × 5%` clamped to `[1 GiB, 16 GiB]`,
            // giving ≈5 GB on a fresh 128 GB Mac (≈1 Gemma-26B entry at
            // F32 per the ~4.8 GB estimate). Operators who want more
            // entries raise the budget via `HF2Q_KV_LCP_RESUME_CAPACITY`
            // (byte-suffix form: e.g. `10g` = 10 GiB; legacy entry-count
            // form `8` still accepted with a deprecation warning).
            //
            // `HF2Q_KV_LCP_RESUME_CAPACITY` env override is honoured:
            // bare integers < 4096 = legacy entry-count × 300 MB;
            // bare integers ≥ 4096 = raw byte count; suffix b/k/m/g =
            // bytes with multiplier.
            lcp_registry: crate::serve::kv_persist::lcp_registry::LcpRegistry::with_byte_budget(
                crate::serve::kv_persist::lcp_registry::default_lcp_byte_budget(),
            ),
            // ADR-017 Phase E.a iter-2: metrics sink wired by
            // `serve::load_engine` AFTER this constructor returns
            // (before `Engine::spawn` moves the loaded model). `None`
            // until then; `record_lcp_probe` calls are gated on this
            // being `Some`.
            kv_metrics_sink: None,
            provenance,
            // ADR-040 Phase C iter-2c (C2c): None until `spawn_with_mode(
            // SlotAware { .. })` provisions per-layer multi-seq KV
            // scaffolds. SerialFifo path NEVER populates this field —
            // legacy `weights.kv_caches` (MlxKvCache, single-seq) remains
            // the live KV state for byte-equivalence (H23 pin).
            multi_seq_kv: None,
            // ADR-040 Phase C iter-C2c-cont (2026-05-30): None until
            // `spawn_with_mode(SlotAware { .. })` provisions the sibling
            // hybrid scaffold via `alloc_multi_seq_hybrid_kv_for_layer`
            // (the production-default KV regime per H10 falsification at
            // §6.1.11). SerialFifo path NEVER populates this field —
            // legacy `weights.kv_caches` remains live for byte-
            // equivalence (H95 pin — sibling to H23).
            multi_seq_kv_hybrid: None,
            // ADR-040 iter-C2c-cont-cont (2026-05-30) — multi-seq dense
            // F32 + legacy 4-bit scaffold siblings.  None at construction
            // (SerialFifo default).  SlotAware spawn arm provisions IFF
            // the respective env-gate is engaged at process start.
            multi_seq_kv_dense: None,
            multi_seq_kv_mlx: None,
        })
    }

    /// **ADR-040 Phase C iter-2c (C2c) + iter-C2c-cont** — provision
    /// per-layer multi-seq KV scaffolds via the A3a
    /// `alloc_hb_kv_for_layer` allocator AND (per iter-C2c-cont) the
    /// sibling A3b `alloc_multi_seq_hybrid_kv_for_layer` allocator,
    /// both with `n_seqs = max_slots`.
    ///
    /// Called by [`Engine::spawn_with_mode`] when
    /// [`EngineMode::SlotAware`] is selected for a Gemma 4 engine. Sets
    /// [`Self::multi_seq_kv`] to `Some(Vec<MultiSeqHbKvBuffers>)`
    /// (`len() == weights.layers.len()`) UNCONDITIONALLY (C2c semantic
    /// preserved verbatim per H94 — the HB-encoded scaffold is the
    /// opt-out path's KV variant and the iter-2A dispatch fork still
    /// consumes it). Iter-C2c-cont ADDITIVE behaviour: sets
    /// [`Self::multi_seq_kv_hybrid`] to `Some(Vec<MultiSeqHybridKvBuffers>)`
    /// IFF `INVESTIGATION_ENV.hybrid_kv == true` (PRODUCTION DEFAULT
    /// since ADR-029 iter-13 per H10 falsification at §6.1.11).
    ///
    /// **Per-layer params** (mirrors the production allocator at
    /// `gemma4/model.rs:1247-1301`):
    /// - `nkv` ← `config.num_kv_heads_for_layer(i)`
    /// - `hd` ← `config.head_dim_for_layer(i)`
    /// - `cap` ← `config.max_position_embeddings` for full-attn,
    ///   `config.sliding_window` for sliding (via A5c
    ///   `layer_type_to_alloc_params` helper)
    /// - `is_ring` ← `!config.is_full_attention(i)` (matches the
    ///   `is_sliding: !is_full` field on legacy `MlxKvCache`)
    ///
    /// Both scaffolds share these per-layer dimensions — the parallel
    /// provisioning loops use IDENTICAL `(nkv, hd, capacity, is_ring)`
    /// quadruples so a future iter-2B kernel-dispatch refactor can
    /// safely consume the HYBRID slice at the same `slot_id.0 * nkv *
    /// capacity * hd * 2`-byte offset the iter-2A-cont HB-encoded path
    /// uses (modulo F16 element-size scaling for the K buffer).
    ///
    /// **Why BOTH variants** (iter-C2c-cont H91 hypothesis): per-request
    /// KV regime is selected INSIDE the model fn via
    /// [`crate::debug::INVESTIGATION_ENV.hybrid_kv`] at call time (see
    /// `forward_prefill_with_soft_tokens_slot_aware`'s dispatch fork
    /// at `src/serve/forward_prefill.rs:2562`). Both scaffolds must
    /// coexist so the dispatch fork hands the appropriate one to the
    /// kernel without spawn-time regime commitment. The HB scaffold is
    /// ALWAYS provisioned because (a) the field already exists in
    /// `GemmaLoadedModel` per C2c §6.1.21 and the H22 / H78 / H85 pins
    /// require its presence; (b) HF2Q_HYBRID_KV=0 operator override
    /// must still hit a populated scaffold (HB-encoded opt-out fallback).
    ///
    /// # Errors
    ///
    /// Returns the first per-layer HB allocator error verbatim
    /// (`anyhow::Error` from `alloc_hb_kv_for_layer`) per C2c. If the
    /// HB phase succeeds AND `INVESTIGATION_ENV.hybrid_kv` is true,
    /// returns the first per-layer HYBRID allocator error verbatim
    /// (`anyhow::Error` from `alloc_multi_seq_hybrid_kv_for_layer`).
    /// `max_slots == 0` is caught at the allocator's pre-flight
    /// (`n_seqs == 0` → typed error per A3a/A3b's invariants) AND at
    /// this fn's entry (defense-in-depth diagnostic).
    pub fn provision_multi_seq_kv_for_slot_aware(&mut self, max_slots: u32) -> Result<()> {
        use crate::inference::models::gemma4::kv_cache::{
            alloc_hb_kv_for_layer, alloc_multi_seq_dense_kv_for_layer,
            alloc_multi_seq_hybrid_kv_for_layer, alloc_multi_seq_mlx_kv_for_layer,
            layer_type_to_alloc_params_per_slot,
        };
        use crate::serve::config::LayerType;

        if max_slots == 0 {
            anyhow::bail!(
                "ADR-040 C2c: provision_multi_seq_kv_for_slot_aware called with \
                 max_slots == 0; spawn_with_mode invariant is max_slots >= 1 \
                 (EngineMode::SlotAware variant enforces this at the API \
                 boundary — caller violated)"
            );
        }
        let dev = self.ctx.device();
        let num_layers = self.weights.layers.len();

        // ── Phase 1: HB-encoded scaffold (C2c §6.1.21 — UNCHANGED) ───
        //
        // Always provisioned per H94: the opt-out (HF2Q_HYBRID_KV=0)
        // dispatch-fork branch in `forward_prefill_with_soft_tokens_
        // slot_aware` (iter-2A) routes through this scaffold; iter-2A-
        // cont kernel-dispatch refactor consumes this field directly.
        let mut multi_seq: Vec<crate::inference::models::gemma4::kv_cache::MultiSeqHbKvBuffers> =
            Vec::with_capacity(num_layers);
        for i in 0..num_layers {
            let hd = self.config.head_dim_for_layer(i);
            let nkv = self.config.num_kv_heads_for_layer(i);
            let is_full = self.config.is_full_attention(i);
            let layer_type = if is_full {
                LayerType::Full
            } else {
                LayerType::Sliding
            };
            let (is_ring, capacity) = layer_type_to_alloc_params_per_slot(
                layer_type,
                self.config.sliding_window,
                self.config.max_position_embeddings,
                max_slots as usize,
            );
            let buf = alloc_hb_kv_for_layer(dev, i, nkv, hd, capacity, is_ring, max_slots)
                .with_context(|| {
                    format!(
                        "ADR-040 C2c: alloc_hb_kv_for_layer L{i} failed for \
                         max_slots={max_slots} (nkv={nkv}, hd={hd}, cap={capacity}, \
                         is_ring={is_ring})"
                    )
                })?;
            multi_seq.push(buf);
        }
        self.multi_seq_kv = Some(multi_seq);

        // ── Phase 2: Hybrid F16-K + TQ-HB-V scaffold (iter-C2c-cont) ──
        //
        // Provisioned IFF the production-default KV regime is engaged.
        // `INVESTIGATION_ENV.hybrid_kv` is a LazyLock read once at
        // process start; matches the env-read discipline at the iter-2A
        // dispatch fork in `forward_prefill.rs:2562` and the legacy
        // alloc site at `forward_prefill.rs:842`. If the env is opted
        // out, we leave `multi_seq_kv_hybrid = None` (H92 negative-arm
        // pin) — the HB-encoded scaffold above is the only live one.
        if crate::debug::INVESTIGATION_ENV.hybrid_kv {
            let mut multi_seq_hybrid: Vec<
                crate::inference::models::gemma4::kv_cache::MultiSeqHybridKvBuffers,
            > = Vec::with_capacity(num_layers);
            for i in 0..num_layers {
                let hd = self.config.head_dim_for_layer(i);
                let nkv = self.config.num_kv_heads_for_layer(i);
                let is_full = self.config.is_full_attention(i);
                let layer_type = if is_full {
                    LayerType::Full
                } else {
                    LayerType::Sliding
                };
                let (is_ring, capacity) = layer_type_to_alloc_params_per_slot(
                    layer_type,
                    self.config.sliding_window,
                    self.config.max_position_embeddings,
                    max_slots as usize,
                );
                let buf = alloc_multi_seq_hybrid_kv_for_layer(
                    dev, i, nkv, hd, capacity, is_ring, max_slots,
                )
                .with_context(|| {
                    format!(
                        "ADR-040 iter-C2c-cont: alloc_multi_seq_hybrid_kv_for_layer \
                         L{i} failed for max_slots={max_slots} (nkv={nkv}, hd={hd}, \
                         cap={capacity}, is_ring={is_ring}) — production-default \
                         hybrid F16-K + TQ-HB-V regime per H10 falsification §6.1.11"
                    )
                })?;
                multi_seq_hybrid.push(buf);
            }
            self.multi_seq_kv_hybrid = Some(multi_seq_hybrid);
        }
        // else: HF2Q_HYBRID_KV=0 → leave multi_seq_kv_hybrid = None
        // (constructor default).  iter-2A dispatch fork's hybrid_kv
        // branch is unreachable in this configuration — the opt-out
        // HB-encoded branch handles the request via multi_seq_kv.

        // ── Phase 3: Dense F32 KV scaffold (iter-C2c-cont-cont, §6.1.46) ──
        //
        // Provisioned IFF the dense F32 env regime is engaged.
        // `INVESTIGATION_ENV.use_dense` is a LazyLock read once at
        // process start; matches the env-read discipline at the
        // iter-2D dispatch fork in `forward_prefill.rs` + the legacy
        // alloc site at `forward_prefill.rs:621-622`.  If the env is
        // off, we leave `multi_seq_kv_dense = None` (H188 negative-arm
        // pin) — the iter-2D dispatch-fork branch is unreachable.
        //
        // Dtype choice mirrors the legacy site: F16 when HF2Q_F16_KV=1,
        // F32 otherwise (the LCP path's KV dtype invariant per ADR-017
        // Phase E.a iter-3.5a).
        if crate::debug::INVESTIGATION_ENV.use_dense {
            let kv_dtype = if crate::debug::INVESTIGATION_ENV.f16_kv {
                mlx_native::DType::F16
            } else {
                mlx_native::DType::F32
            };
            let mut multi_seq_dense: Vec<
                crate::inference::models::gemma4::kv_cache::MultiSeqDenseKvBuffers,
            > = Vec::with_capacity(num_layers);
            for i in 0..num_layers {
                let hd = self.config.head_dim_for_layer(i);
                let nkv = self.config.num_kv_heads_for_layer(i);
                let is_full = self.config.is_full_attention(i);
                let layer_type = if is_full {
                    LayerType::Full
                } else {
                    LayerType::Sliding
                };
                let (is_ring, capacity) = layer_type_to_alloc_params_per_slot(
                    layer_type,
                    self.config.sliding_window,
                    self.config.max_position_embeddings,
                    max_slots as usize,
                );
                let buf = alloc_multi_seq_dense_kv_for_layer(
                    dev, i, nkv, hd, capacity, is_ring, kv_dtype, max_slots,
                )
                .with_context(|| {
                    format!(
                        "ADR-040 iter-C2c-cont-cont: alloc_multi_seq_dense_kv_for_layer \
                         L{i} failed for max_slots={max_slots} (nkv={nkv}, hd={hd}, \
                         cap={capacity}, is_ring={is_ring}, dtype={:?}) — \
                         HF2Q_USE_DENSE=1 opt-in pre-default surface per §6.1.46",
                        kv_dtype,
                    )
                })?;
                multi_seq_dense.push(buf);
            }
            self.multi_seq_kv_dense = Some(multi_seq_dense);
        }
        // else: HF2Q_USE_DENSE=0 → leave multi_seq_kv_dense = None.

        // ── Phase 4: Legacy 4-bit MlxKvCache scaffold (iter-C2c-cont-cont, §6.1.46) ──
        //
        // Provisioned IFF HF2Q_TQ_CODEBOOK_BITS=4 is engaged.  Mirror
        // of Phase 1's env-read discipline applied to the cb_bits gate.
        // `norms_per_pos = (hd / 256).max(1)` matches the legacy alloc
        // site at `gemma4/model.rs:1273`.
        let cb_bits_provision: u32 = match std::env::var("HF2Q_TQ_CODEBOOK_BITS").as_deref() {
            Ok("4") => 0,
            Ok("5") => 5,
            Ok("6") => 6,
            Ok("8") => 8,
            _ => 8, // DEFAULT: 8-bit (matches forward_prefill.rs line 835)
        };
        if cb_bits_provision == 0 {
            let mut multi_seq_mlx: Vec<
                crate::inference::models::gemma4::kv_cache::MultiSeqMlxKvCache,
            > = Vec::with_capacity(num_layers);
            for i in 0..num_layers {
                let hd = self.config.head_dim_for_layer(i);
                let nkv = self.config.num_kv_heads_for_layer(i);
                let is_full = self.config.is_full_attention(i);
                let layer_type = if is_full {
                    LayerType::Full
                } else {
                    LayerType::Sliding
                };
                let (is_ring, capacity) = layer_type_to_alloc_params_per_slot(
                    layer_type,
                    self.config.sliding_window,
                    self.config.max_position_embeddings,
                    max_slots as usize,
                );
                let norms_per_pos = (hd / 256).max(1);
                let buf = alloc_multi_seq_mlx_kv_for_layer(
                    dev,
                    i,
                    nkv,
                    hd,
                    capacity,
                    is_ring,
                    norms_per_pos,
                    max_slots,
                )
                .with_context(|| {
                    format!(
                        "ADR-040 iter-C2c-cont-cont: alloc_multi_seq_mlx_kv_for_layer \
                         L{i} failed for max_slots={max_slots} (nkv={nkv}, hd={hd}, \
                         cap={capacity}, is_ring={is_ring}, norms_per_pos={norms_per_pos}) \
                         — HF2Q_TQ_CODEBOOK_BITS=4 opt-in pre-default surface per §6.1.46"
                    )
                })?;
                multi_seq_mlx.push(buf);
            }
            self.multi_seq_kv_mlx = Some(multi_seq_mlx);
        }
        // else: HF2Q_TQ_CODEBOOK_BITS != "4" → leave multi_seq_kv_mlx = None.

        Ok(())
    }
}

impl LoadInfoBuilder for GemmaLoadedModel {
    fn build_load_info(
        &self,
        gguf: &mlx_native::gguf::GgufFile,
        load_wall_clock: Duration,
        kv_cache_budget_bytes: Option<u64>,
        kv_spill_active: bool,
    ) -> LoadInfo {
        let arch_str = load_info::arch_str_from_gguf(gguf);
        let moe = if self.config.num_experts > 0 && self.config.top_k_experts > 0 {
            Some(MoeShape {
                n_experts: self.config.num_experts as u32,
                n_experts_per_tok: self.config.top_k_experts as u32,
            })
        } else {
            None
        };

        LoadInfo {
            model_id: self.model_id.clone(),
            arch_str,
            arch_family: ArchFamily::Gemma4,
            model_path: self.model_path.clone(),
            on_disk_bytes: load_info::on_disk_bytes(&self.model_path),
            backend_chip: self.ctx.gpu_name(),
            backend: "mlx-native",
            n_layers: self.config.num_hidden_layers as u32,
            hidden_size: self.config.hidden_size as u32,
            vocab_size: self.config.vocab_size as u32,
            n_attention_heads: self.config.num_attention_heads as u32,
            n_key_value_heads: self.config.num_key_value_heads as u32,
            head_dim: self.config.head_dim as u32,
            sliding_window: Some(self.config.sliding_window as u32),
            // Iter-82 fix: report the actual full-attention interval
            // computed from layer_types (default for gemma is every 6th
            // layer = Full). Pre-iter-82 this was hardcoded `None`,
            // misleading the load banner to say "full_attn_every=none"
            // when gemma actually has 5 of 30 Full-attention layers.
            full_attention_interval: self.config.full_attention_interval(),
            max_context_length: self.context_length.map(|v| v as u32),
            moe,
            quant_label: self.quant_type.clone(),
            quant_bpw: load_info::compute_bpw(gguf),
            tokenizer_source: TokenizerSource::HfTokenizerJson {
                path: self.tokenizer_path.clone(),
            },
            eos_token_ids: self.eos_token_ids.clone(),
            bos_token_id: gguf.metadata_u32("tokenizer.ggml.bos_token_id"),
            chat_template_source: if gguf.metadata_string("tokenizer.chat_template").is_some() {
                ChatTemplateSource::GgufEmbedded
            } else {
                ChatTemplateSource::HardcodedFallback {
                    name: "FALLBACK_GEMMA4_API_CHAT_TEMPLATE",
                }
            },
            provenance: self.provenance.clone(),
            vision_projector: None,
            load_wall_clock,
            resident_weight_bytes: None,
            kv_cache_budget_bytes,
            kv_spill_active,
            // 2026-05-23: surface Gemma 4 TQ-active state on the load
            // banner. On Gemma 4, TQ-default-on is the production path
            // (ADR-007 Path C, closed 2026-04-24; followup ADR's Gate H
            // failure on non-DWQ models was closed 2026-05-23 with
            // cosine_mean 0.999843 on APEX). Inactive only when the
            // operator forces dense via `HF2Q_USE_DENSE=1` or
            // `HF2Q_LAYER_POLICY=dense_all`. Matches the per-family
            // banner text in `load_info::emit_text`.
            tq_kv_active: !crate::debug::investigation_env::INVESTIGATION_ENV.use_dense
                && !matches!(
                    crate::debug::investigation_env::INVESTIGATION_ENV
                        .layer_policy
                        .as_deref(),
                    Some("dense_all")
                ),
            // ADR-040 §3.5 iter-A5c (cfa-A5b CRITICAL #1) — Gemma 4
            // layers are heterogeneous in KV-cache shape: sliding
            // layers carry `num_key_value_heads × head_dim` (canonical
            // 8 × 256) while full-attention layers carry
            // `num_global_key_value_heads × global_head_dim`
            // (canonical 2 × 512). The flattened scalar formula on
            // `LoadInfo::kv_bytes_per_token` (n_layers × n_kv_heads ×
            // head_dim × dtype_bytes × 2) over-counts by ~9% for
            // canonical 30-layer 27B (61_440 elem vs exact 56_320 elem
            // per token) — a safe upper bound that false-rejects
            // borderline requests, never under-counts. iter-A5c replaces
            // the over-count with the EXACT per-layer sum so the engine
            // seam admit-time check matches the actual KV allocation
            // shape (`gemma4/model.rs:1247-1257`:
            // `head_dim_for_layer(i) * num_kv_heads_for_layer(i) *
            // capacity` per layer, with capacity differing between
            // sliding_window and max_position_embeddings — capacity is
            // the per-token multiplier that lives in
            // `kv_bytes_for_request`, the SHAPE is what differs per
            // layer and must be summed here).
            kv_bytes_per_token_override: Some(load_info::gemma4_exact_kv_bytes_per_token(
                &self.config,
            )),
        }
    }
}

impl LoadInfoBuilder for LoadedModel {
    fn build_load_info(
        &self,
        gguf: &mlx_native::gguf::GgufFile,
        load_wall_clock: Duration,
        kv_cache_budget_bytes: Option<u64>,
        kv_spill_active: bool,
    ) -> LoadInfo {
        match self {
            LoadedModel::Gemma(g) => g.build_load_info(
                gguf,
                load_wall_clock,
                kv_cache_budget_bytes,
                kv_spill_active,
            ),
            LoadedModel::Qwen35(q) => q.build_load_info(
                gguf,
                load_wall_clock,
                kv_cache_budget_bytes,
                kv_spill_active,
            ),
            LoadedModel::Qwen3VlText(v) => v.build_load_info(
                gguf,
                load_wall_clock,
                kv_cache_budget_bytes,
                kv_spill_active,
            ),
            LoadedModel::Deepseek4(d) => d.build_load_info(
                gguf,
                load_wall_clock,
                kv_cache_budget_bytes,
                kv_spill_active,
            ),
        }
    }
}

// ---------------------------------------------------------------------------
// Engine::spawn and worker loop
// ---------------------------------------------------------------------------

impl Engine {
    /// Spawn the worker thread and return a handle. The `queue_capacity` sets
    /// the mpsc channel buffer; when full, handlers receive a `queue_full`
    /// error and map it to 429 + Retry-After (Decision #19).
    pub fn spawn(
        loaded: LoadedModel,
        queue_capacity: usize,
        kv_cache_budget_bytes: Option<u64>,
    ) -> Self {
        let (tx, rx) = mpsc::channel::<Request>(queue_capacity.max(1));

        // Iter-215 Wedge-2: accessor methods replace flat-struct field
        // reads.  The enum dispatches per-variant; the EngineInner
        // metadata fields (model_id, hidden_size, etc.) are populated
        // identically for both Gemma and Qwen35 variants.
        let model_id = loaded.model_id().to_string();
        let context_length = loaded.context_length();
        let quant_type = loaded.quant_type().map(|s| s.to_string());
        let hidden_size = loaded.hidden_size();
        let vocab_size = loaded.vocab_size();
        let eos_token_ids = loaded.eos_token_ids().to_vec();
        let tokenizer = Arc::new(loaded.tokenizer().clone());
        let chat_template = Arc::new(loaded.chat_template().to_string());
        let arch = match &loaded {
            LoadedModel::Gemma(_) => LoadedArch::Gemma,
            LoadedModel::Qwen35(_) => LoadedArch::Qwen35,
            LoadedModel::Qwen3VlText(_) => LoadedArch::Qwen3VlText,
            LoadedModel::Deepseek4(_) => LoadedArch::Deepseek4,
        };

        // Phase B-dense.2 follow-up: snapshot the KV-spill shape
        // descriptor BEFORE moving `loaded` into the worker thread.
        // Read-only on `MlxModelWeights` — only iterates per-layer
        // shape fields. `None` for the Qwen35 variant.
        let kv_spill_descriptor: Option<super::kv_spill_descriptor::KvSpillDescriptor> =
            match &loaded {
                LoadedModel::Gemma(g) => {
                    // Read the operator-time HF2Q_F16_KV env via the same
                    // `INVESTIGATION_ENV` LazyLock that
                    // `forward_prefill.rs:259` reads — so the descriptor
                    // matches what the prefill allocator will pick. Default
                    // is F32 (no env set / =0).
                    let kv_dtype = if crate::debug::INVESTIGATION_ENV.f16_kv {
                        super::kv_spill_descriptor::KvDType::F16
                    } else {
                        super::kv_spill_descriptor::KvDType::F32
                    };
                    // Static spill-side budget. Mirrors `SamplingParams`
                    // default (`max_tokens: 512`) — the hook's
                    // restore-before-prefill path uses this to seed
                    // full-attention layer linear capacity, then
                    // `forward_prefill.rs:274-285` reallocates to
                    // `seq_len + max_tokens` per request.
                    let max_decode_tokens = 512usize;
                    // ADR-017 §F4: lift provenance bits from the loader
                    // (captured at GGUF-open time) and the loaded chat
                    // template (sha256-hashed) into the descriptor.
                    // Foreign GGUFs yield Provenance::External →
                    // KvSpillProvenance::default() (all-empty); the
                    // spiller's family_model_fp falls back to the
                    // legacy `(repo, quant, "", "", "")` namespace.
                    let provenance = match &g.provenance {
                        crate::core::provenance::Provenance::Hf2q {
                            producer_version,
                            source_sha256,
                            ..
                        } => super::kv_spill_descriptor::KvSpillProvenance {
                            producer_version: producer_version.clone(),
                            source_sha256: source_sha256.clone(),
                            tokenizer_chat_template_hash:
                                super::kv_spill_descriptor::KvSpillProvenance::hash_chat_template(
                                    &g.chat_template,
                                ),
                        },
                        crate::core::provenance::Provenance::External => {
                            super::kv_spill_descriptor::KvSpillProvenance::default()
                        }
                    };
                    Some(
                        super::kv_spill_descriptor::KvSpillDescriptor::from_gemma_loaded_model(
                            &g.weights,
                            max_decode_tokens,
                            kv_dtype,
                            provenance,
                        ),
                    )
                }
                LoadedModel::Qwen35(_) => None,
                // iter-228a: Qwen3-VL text LM doesn't yet wire a
                // KV-spill descriptor; the chat arm short-circuits to
                // 501 before any KV-spill code path runs. iter-228b's
                // forward wiring will revisit (mirror the gemma branch
                // shape against the dense Qwen3-VL KV cache).
                LoadedModel::Qwen3VlText(_) => None,
                LoadedModel::Deepseek4(_) => None,
            };

        // **ADR-017 §B-tq.4 iter-4** — capture the per-layer TQ-active
        // shape descriptor when `HF2Q_TQ_KV=1` AND the loaded model
        // exposes per-layer `kv_caches[i].k_packed`.  Used by
        // `TqPackedSpillFactory::try_construct` to build a per-layer-
        // correct cfg instead of the cmd_serve fallback.
        let tq_packed_descriptor: Option<super::tq_packed_descriptor::TqPackedSpillDescriptor> =
            if super::tq_packed_descriptor::is_tq_active_mode() {
                match &loaded {
                    LoadedModel::Gemma(g) => {
                        let provenance_for_tq = match &g.provenance {
                            crate::core::provenance::Provenance::Hf2q {
                                producer_version,
                                source_sha256,
                                ..
                            } => super::kv_spill_descriptor::KvSpillProvenance {
                                producer_version: producer_version.clone(),
                                source_sha256: source_sha256.clone(),
                                tokenizer_chat_template_hash:
                                    super::kv_spill_descriptor::KvSpillProvenance::hash_chat_template(
                                        &g.chat_template,
                                    ),
                            },
                            crate::core::provenance::Provenance::External => {
                                super::kv_spill_descriptor::KvSpillProvenance::default()
                            }
                        };
                        super::tq_packed_descriptor::TqPackedSpillDescriptor::from_gemma_loaded_model_tq(
                            &g.weights,
                            provenance_for_tq,
                        )
                    }
                    // TQ-active KV-persist is Gemma-4-only at this iter
                    // (per family-scoping discipline established by B-dense.1).
                    LoadedModel::Qwen35(_)
                    | LoadedModel::Qwen3VlText(_)
                    | LoadedModel::Deepseek4(_) => None,
                }
            } else {
                None
            };

        let kv_spill_active = kv_spill_descriptor.is_some();
        let gguf = mlx_native::gguf::GgufFile::open(loaded.model_path())
            .expect("re-open loaded GGUF for Engine load-info snapshot");
        let info = Arc::new(loaded.build_load_info(
            &gguf,
            loaded.load_duration(),
            kv_cache_budget_bytes,
            kv_spill_active,
        ));

        let registration = super::registry::find_for(&model_id);
        if let Some(ref r) = registration {
            tracing::info!(
                family = r.family,
                reasoning = r.has_reasoning(),
                tools = r.has_tools(),
                "hf2q-engine: matched model registration"
            );
        } else {
            tracing::info!(
                model_id = %model_id,
                "hf2q-engine: no matching model registration (text emitted as plain content)"
            );
        }

        // ADR-040 Phase C iter-2a (C2b) — the 3-arg `spawn` is the
        // ADR-005 byte-equivalence entry point; the worker is spawned
        // under `EngineMode::SerialFifo` with `max_slots = 1`. The
        // `queue_capacity` value reaches `worker_run` so the
        // `FifoSchedulerAdapter` constructed at thread entry mirrors the
        // mpsc channel's `queue_capacity.max(1)` cap (dossier §2.3 + §4
        // iter-2a step 3). The pre-seeded `SchedulerStats` mirrors what
        // `FifoSchedulerAdapter::stats()` returns on a fresh adapter so
        // a /metrics scrape between spawn and the first admit reports
        // sensible defaults (zero counters, configured capacity).
        let queue_capacity_u32 = queue_capacity.max(1) as u32;
        let initial_mode = EngineMode::SerialFifo;
        let initial_max_slots: u32 = 1;
        // ADR-040 §3.5 iter-A5b — per-slot KV byte budget. Under
        // SerialFifo `max_slots = 1` so `kv_cache_budget_bytes / 1`
        // = `kv_cache_budget_bytes`; `None` ⇒ `0` ⇒ enforcement
        // disabled (pre-A5 byte-equivalence preserved for operators
        // who do not set `--kv-cache-budget-bytes`). The same field
        // also flows to the worker thread so the scheduler-side
        // FifoSchedulerAdapter::new_with_kv_budget enforces at admit.
        let initial_per_slot_kv_budget_bytes: u64 = kv_cache_budget_bytes
            .map(|b| b / u64::from(initial_max_slots.max(1)))
            .unwrap_or(0);
        let initial_kv_bytes_per_token_cached: u64 = info.kv_bytes_per_token();
        let scheduler_stats_snapshot = Arc::new(Mutex::new(SchedulerStats {
            policy: SchedulerPolicy::FifoSerial,
            in_flight_slots: 0,
            queue_capacity: queue_capacity_u32,
            admitted_total: 0,
            rejected_429_total: 0,
            completed_total: 0,
        }));
        let worker_stats_handle = Arc::clone(&scheduler_stats_snapshot);

        // Move registration into the worker closure in addition to the handle.
        let worker_registration = registration.clone();
        let worker_per_slot_budget = initial_per_slot_kv_budget_bytes;
        let worker_kv_bytes_per_token = initial_kv_bytes_per_token_cached;
        let worker_handle = std::thread::Builder::new()
            .name("hf2q-engine".into())
            .spawn(move || {
                worker_run(
                    loaded,
                    rx,
                    worker_registration,
                    initial_mode,
                    queue_capacity_u32,
                    worker_stats_handle,
                    worker_per_slot_budget,
                    worker_kv_bytes_per_token,
                )
            })
            .expect("spawn hf2q-engine thread");

        Engine {
            inner: Arc::new(EngineInner {
                tx,
                worker_handle: Mutex::new(Some(worker_handle)),
                info,
                arch,
                model_id,
                context_length,
                quant_type,
                hidden_size,
                vocab_size,
                eos_token_ids,
                tokenizer,
                chat_template,
                registration,
                token_bytes: std::sync::OnceLock::new(),
                kv_spill_descriptor,
                tq_packed_descriptor,
                mode: initial_mode,
                max_slots: initial_max_slots,
                per_slot_kv_budget_bytes: initial_per_slot_kv_budget_bytes,
                kv_bytes_per_token_cached: initial_kv_bytes_per_token_cached,
                scheduler_stats_snapshot,
            }),
        }
    }

    /// ADR-040 Phase C iter-1.5 — `spawn` with explicit mode selection.
    ///
    /// At iter-1.5, only [`EngineMode::SerialFifo`] survives validation —
    /// it delegates to [`Engine::spawn`] and returns `Ok(...)`. The
    /// [`EngineMode::SlotAware`] variant is rejected with
    /// [`EngineSpawnError::ModeNotYetWired`] at the API boundary; Phase C
    /// iter-2 will replace this rejection with the live `Scheduler` +
    /// `MultiSeqKvCache` path.
    ///
    /// The existing [`Engine::spawn`] remains the production entry point and
    /// is byte-equivalent to calling
    /// `spawn_with_mode(loaded, queue_capacity, kv_cache_budget_bytes, EngineMode::SerialFifo)`.
    ///
    /// # Why this is `Result` at iter-1.5
    ///
    /// Per ADR-040 §7 "no fallback, no stub (todo later) code", silently
    /// discarding the requested mode (as iter-1 did with
    /// `let _ = mode;`) is a Liskov-substitution violation: a caller
    /// passing `SlotAware{max_slots:8}` and then reading `engine.mode()`
    /// would get back `SerialFifo`. Both adversarial reviewers (Codex +
    /// Claude) flagged this as CRITICAL in the iter-1.5 review. The
    /// `Result` return + typed `EngineSpawnError` makes the iter-1 vs
    /// iter-2 cliff visible at the type level: today you MUST handle the
    /// `Err` arm, and iter-2 makes both arms `Ok`.
    ///
    /// # Errors
    ///
    /// Returns [`EngineSpawnError::ModeNotYetWired`] when called with
    /// [`EngineMode::SlotAware`] at iter-1.5. Iter-2 lands the
    /// `Scheduler` runtime and the rejection is removed.
    pub fn spawn_with_mode(
        loaded: LoadedModel,
        queue_capacity: usize,
        kv_cache_budget_bytes: Option<u64>,
        mode: EngineMode,
    ) -> std::result::Result<Self, EngineSpawnError> {
        match mode {
            EngineMode::SerialFifo => {
                // Delegate to the byte-equivalence entry point and then
                // overwrite the stored mode to echo the caller's request.
                // (The 3-arg `spawn` already initializes mode to
                // SerialFifo, so this is a no-op for this variant — but
                // we go through the assignment for symmetry with iter-2's
                // SlotAware path, which will store the requested
                // max_slots on EngineInner.)
                let mut engine = Self::spawn(loaded, queue_capacity, kv_cache_budget_bytes);
                // Safe: this is the only `Arc` to the freshly-spawned
                // `EngineInner` at this point — `spawn` has not handed
                // out any clones yet. `get_mut` returns `Some(&mut ...)`
                // for the SerialFifo arm and we set the mode field.
                if let Some(inner) = Arc::get_mut(&mut engine.inner) {
                    inner.mode = EngineMode::SerialFifo;
                } else {
                    // Defensive: spawn invariant says the Arc has refcount
                    // 1 here. If a future refactor breaks that, this is
                    // an unreachable branch — but better to assert than
                    // silently drop the mode write.
                    debug_assert!(
                        false,
                        "ADR-040 invariant: freshly-spawned Engine inner Arc must have refcount 1"
                    );
                }
                Ok(engine)
            }
            // ADR-040 Phase C iter-2c (C2c) — per-arch SlotAware dispatch.
            //
            // Gemma 4 lands `Ok(Engine)` via Path B: spawn succeeds with
            // structurally-provisioned multi-seq KV scaffolds (per-layer
            // `MultiSeqHbKvBuffers` allocated via the A3a allocator with
            // `n_seqs = max_slots`); the worker thread runs the existing
            // single-seq forward path at SlotId(0) and surfaces typed
            // `MultiSeqError::CapabilityUnsupported` for SlotId(N>0)
            // admissions — kernel-level slot-offset routing through
            // `forward_prefill.rs` is iter-C2c-cont (gated on B4c per
            // ADR-040 §6 + §6.1.21).
            //
            // Qwen35 + Qwen3VlText keep the typed `ModeNotYetWired`
            // rejection until their respective per-arch worker arms ship
            // (C2d for Qwen35; future iter for Qwen3VlText). The C2 dossier
            // §2.4 scope split is honoured: each arch's lift is
            // independent.
            EngineMode::SlotAware { max_slots } => {
                // ADR-040 Phase F (2026-06-24) — continuous-batching
                // capacity gate.  Default 8 (`HF2Q_MAX_BATCHED_SLOTS`,
                // `ADR040_F_DEFAULT_CONTINUOUS_BATCHING_MAX_SLOTS`) — the
                // upper edge of the §6.1.53/54 dossier's 4-8 safe zone,
                // empirically validated for the WIRED continuous-batching
                // path (byte-identical + coherence-proven at N=8; KV-mem
                // bounded ~4.2 GB @ 8×32k).  This is DECOUPLED from the
                // spec-decode drafter gate
                // (`ADR040_A4_DEFAULT_SPEC_DECODE_MAX_BATCHED_SLOTS`, still
                // 4, fail-closed) per codex's `b671dfe0` review item (c):
                // the unwired drafter (§6.1.55-F5 API-scaffold) MUST NOT
                // inherit the relaxed continuous default when it lands.
                // `HF2Q_SPEC_DECODE_ALLOW_OVERSIZED=1` is the explicit
                // opt-in past the ceiling — silent capping would be a
                // Liskov-substitution violation per ADR-040 §7 no-fallback
                // mantra (H229 pins this).
                //
                // Gate sits BEFORE per-arch dispatch so the policy is
                // arch-uniform: Gemma 4 / Qwen35 / Qwen3VL all surface
                // the SAME typed error at the SAME `max_slots`
                // threshold, regardless of which per-arch provisioner
                // would have run.
                let threshold = read_continuous_batching_max_slots(|name| std::env::var(name).ok());
                let allow_oversized =
                    read_spec_decode_allow_oversized(|name| std::env::var(name).ok());
                if max_slots > threshold && !allow_oversized {
                    return Err(EngineSpawnError::SpecDecodeMaxSlotsAboveBatchedThreshold {
                        max_slots,
                        threshold,
                        cite: ADR040_A4_DOSSIER_CITE,
                    });
                }
                Self::spawn_with_mode_slot_aware_arch_dispatch(
                    loaded,
                    queue_capacity,
                    kv_cache_budget_bytes,
                    max_slots,
                )
            }
        }
    }

    /// ADR-040 Phase A4 iter-1 (2026-05-30) — per-arch dispatch helper
    /// extracted from [`Self::spawn_with_mode`] so the spec-decode
    /// oversized-slots threshold gate (above) sits at a single
    /// arch-uniform point.  Body is byte-for-byte the prior per-arch
    /// `match loaded` block; only the surrounding control flow lifted.
    ///
    /// **Invariant**: this is reached ONLY after the threshold gate
    /// passes (either `max_slots <= threshold` or
    /// `HF2Q_SPEC_DECODE_ALLOW_OVERSIZED=1`).  Per-arch handlers retain
    /// their own `max_slots == 0` defensive checks for the
    /// `ModeNotYetWired { iter_required: "caller bug ..." }` mantra
    /// (pre-iter-A4 contract preserved verbatim).
    fn spawn_with_mode_slot_aware_arch_dispatch(
        loaded: LoadedModel,
        queue_capacity: usize,
        kv_cache_budget_bytes: Option<u64>,
        max_slots: u32,
    ) -> std::result::Result<Self, EngineSpawnError> {
        match loaded {
                LoadedModel::Gemma(mut g) => {
                    if max_slots == 0 {
                        // EngineMode::SlotAware is constructed by callers
                        // who guarantee max_slots >= 1; this is a defensive
                        // rejection at the API boundary so a `max_slots = 0`
                        // never reaches `provision_multi_seq_kv_for_slot_aware`.
                        return Err(EngineSpawnError::ModeNotYetWired {
                            iter_landed: "C2c",
                            iter_required: "caller bug: EngineMode::SlotAware with max_slots == 0 \
                                            — require max_slots >= 1",
                        });
                    }
                    // Provision per-layer multi-seq KV scaffolds BEFORE
                    // moving the loaded model into the worker thread; if
                    // provisioning fails, surface the typed error before
                    // any thread is spawned.
                    //
                    // iter-C2c-cont (2026-05-30): two phases provision
                    // BOTH the HB-encoded scaffold (always) AND the
                    // hybrid F16-K + TQ-HB-V scaffold (when
                    // HF2Q_HYBRID_KV=1 — the production default per H10
                    // falsification §6.1.11). On allocator failure we
                    // distinguish which regime failed by inspecting the
                    // `multi_seq_kv` field's populated state: if it's
                    // still None, the HB phase (Phase 1) failed; if it
                    // IS populated and we still got an error, the
                    // hybrid phase (Phase 2) failed. Both surfaces emit
                    // their own typed variant for operator-grep
                    // disambiguation.
                    if let Err(e) = g.provision_multi_seq_kv_for_slot_aware(max_slots) {
                        // Hybrid phase failure: HB phase already
                        // populated `multi_seq_kv` (Phase 1 invariant).
                        // Surface the iter-C2c-cont-named typed error
                        // distinct from the C2c HB-phase variant so
                        // operator log greps + per-regime debug paths
                        // route to the right pin pointer.
                        if g.multi_seq_kv.is_some() {
                            return Err(
                                EngineSpawnError::Gemma4HybridSlotAwareProvisionFailed {
                                    max_slots,
                                    cause: e.to_string(),
                                },
                            );
                        }
                        // HB phase failure: same shape as pre-iter-C2c-
                        // cont. Preserves H22_gemma4_spawn_fail_variant_
                        // carries_max_slots_and_cause + the C2c error-
                        // surface contract verbatim.
                        return Err(EngineSpawnError::Gemma4SlotAwareProvisionFailed {
                            max_slots,
                            cause: e.to_string(),
                        });
                    }
                    let loaded = LoadedModel::Gemma(g);
                    let engine = Self::spawn_inner_with_slot_aware(
                        loaded,
                        queue_capacity,
                        kv_cache_budget_bytes,
                        max_slots,
                    );
                    Ok(engine)
                }
                LoadedModel::Qwen35(mut q) => {
                    // ADR-040 Phase C iter-2d (C2d) — Qwen35 SlotAware
                    // engine activation. Mirrors the Gemma 4 arm above:
                    // spawn-time multi-seq KV provisioning via the A2a
                    // `HybridKvCache::new(.., n_seqs = max_slots)`
                    // allocator; the worker thread still serves SlotId(0)
                    // through the existing single-seq forward path and
                    // surfaces typed `MultiSeqError::Capability
                    // Unsupported` for SlotId(N>0) admissions —
                    // kernel-level slot routing through the prompt-cache
                    // restore + spec-decode + hybrid persistor surfaces
                    // is iter-C2d-cont (gated on R4 + R4-bis per
                    // ADR-040 §6 + §6.1.22). B4b (decode-side slot
                    // threading) already shipped 2026-05-24 — the
                    // forward-path kernels accept SlotId(N>0) but the
                    // per-request `alloc_kv_cache_for_request` allocates
                    // `n_seqs=1` HybridKvCache instances per call (the
                    // persistent multi-seq cache is provisioned here as
                    // scaffolding; the worker hot path uses it once
                    // iter-C2d-cont lifts the per-request alloc into
                    // the persistent cache + slot-aware prompt-cache
                    // restore lands).
                    if max_slots == 0 {
                        // Defensive: EngineMode::SlotAware callers
                        // guarantee max_slots >= 1, but pin the boundary
                        // so the provisioner never sees zero.
                        return Err(EngineSpawnError::ModeNotYetWired {
                            iter_landed: "C2d",
                            iter_required: "caller bug: EngineMode::SlotAware with max_slots == 0 \
                                            — require max_slots >= 1",
                        });
                    }
                    if let Err(e) = q.provision_multi_seq_kv_for_slot_aware(max_slots) {
                        // Wrap into a typed spawn error so callers can
                        // distinguish "Qwen35 SlotAware provisioning
                        // failed" from "ModeNotYetWired" without
                        // string-matching anyhow.
                        return Err(EngineSpawnError::Qwen35SlotAwareProvisionFailed {
                            max_slots,
                            cause: e.to_string(),
                        });
                    }
                    let loaded = LoadedModel::Qwen35(q);
                    let engine = Self::spawn_inner_with_slot_aware(
                        loaded,
                        queue_capacity,
                        kv_cache_budget_bytes,
                        max_slots,
                    );
                    Ok(engine)
                }
                LoadedModel::Qwen3VlText(mut v) => {
                    // ADR-040 Phase C iter-C2e (2026-05-30) — Qwen3-VL
                    // SlotAware engine activation. Direct mirror of
                    // C2c §6.1.21 (Gemma 4) + C2d §6.1.22 (Qwen35) for
                    // the Qwen3-VL text-LM family.  Pre-C2e this arm
                    // returned `ModeNotYetWired { iter_required: "C2e
                    // (...)"}`; iter-C2e flips it to `Ok(Engine)` via
                    // Path B: spawn-time witness provisioning + worker
                    // arms typed-clamped at SlotId(N>0).
                    //
                    // Path B (witness + typed worker-arm deferral)
                    // chosen because Qwen3-VL today runs the iter-9b
                    // naive O(N²) re-prefill loop in
                    // `engine_qwen3vl::generate_qwen3vl_text_once`
                    // with no persistent KV cache; the real per-step
                    // KV cache is upstream-blocked on iter-228a (the
                    // 501-sentinel arms of `worker_run` for streaming
                    // / embed / vision-augmented requests).
                    // Path A (full activation with worker hot-path
                    // routing through a persistent cache) is gated on
                    // iter-228a + iter-C2e-cont landing — those iters
                    // ship the persistent KV scaffold + the four
                    // worker-arm lift mirrors of C2d-cont-kernel
                    // iter-{1,2,3,4} §6.1.27-6.1.30 for the Qwen3-VL
                    // architecture.
                    //
                    // The four worker arms (Generate / GenerateStream /
                    // Embed / GenerateWithSoftTokens) surface typed
                    // `MultiSeqError::CapabilityUnsupported` at
                    // SlotId(N>0) with an operator-grep'able label
                    // naming `iter-C2e-cont per ADR-040 §6.1.52`
                    // (post iter-228a worker-hot-path lift) AND
                    // `iter-228a` (the upstream-blocker for the
                    // persistent KV cache itself).
                    if max_slots == 0 {
                        // Defensive: EngineMode::SlotAware callers
                        // guarantee max_slots >= 1, but pin the
                        // boundary so the provisioner never sees zero.
                        return Err(EngineSpawnError::ModeNotYetWired {
                            iter_landed: "C2e",
                            iter_required: "caller bug: EngineMode::SlotAware with max_slots == 0 \
                                            — require max_slots >= 1",
                        });
                    }
                    if let Err(e) = v.provision_multi_seq_kv_for_slot_aware(max_slots) {
                        // Wrap into a typed spawn error so callers can
                        // distinguish "Qwen3-VL SlotAware provisioning
                        // failed" from "ModeNotYetWired" without
                        // string-matching anyhow. Mirrors C2d's
                        // Qwen35SlotAwareProvisionFailed shape.
                        return Err(EngineSpawnError::Qwen3VLSlotAwareProvisionFailed {
                            max_slots,
                            cause: e.to_string(),
                        });
                    }
                    let loaded = LoadedModel::Qwen3VlText(v);
                    let engine = Self::spawn_inner_with_slot_aware(
                        loaded,
                        queue_capacity,
                        kv_cache_budget_bytes,
                        max_slots,
                    );
                    Ok(engine)
                }
                LoadedModel::Deepseek4(_) => Err(EngineSpawnError::ModeNotYetWired {
                    iter_landed: "deepseek4-agentic-serving",
                    iter_required: "DeepSeek-V4 currently supports Legacy single-session mode; \
                                    SlotAware scheduling needs per-slot recurrent and compressed-KV state",
                }),
        }
    }

    /// **ADR-040 Phase C iter-2c (C2c)** — internal helper mirroring
    /// `Engine::spawn` but storing the requested SlotAware mode +
    /// max_slots on `EngineInner` and constructing the
    /// `InflightBatchedScheduler` (instead of `FifoSchedulerAdapter`)
    /// at worker entry.
    ///
    /// Per the C2 dossier §2.7 R2: under Shape A the worker still
    /// processes one request at a time. The InflightBatchedScheduler
    /// admits up to `max_slots` distinct `SlotId`s — at iter-C2c the
    /// worker only routes SlotId(0) through the single-seq forward
    /// path; SlotId(N>0) returns
    /// `MultiSeqError::CapabilityUnsupported` per the typed deferral
    /// for kernel-level routing (iter-C2c-cont). This preserves
    /// scheduler-level multi-slot semantics (admit can hand out
    /// distinct slot IDs) while honoring the kernel-side limitation.
    ///
    /// Mirrors `Engine::spawn` step-by-step at the field-init level —
    /// the duplication is intentional per the dossier §2.6 "the
    /// cleanest move is to factor the spawn body into a private helper
    /// that both entry points call with their respective mode value"
    /// observation; this is that private helper for the SlotAware path.
    fn spawn_inner_with_slot_aware(
        loaded: LoadedModel,
        queue_capacity: usize,
        kv_cache_budget_bytes: Option<u64>,
        max_slots: u32,
    ) -> Self {
        let (tx, rx) = mpsc::channel::<Request>(queue_capacity.max(1));

        let model_id = loaded.model_id().to_string();
        let context_length = loaded.context_length();
        let quant_type = loaded.quant_type().map(|s| s.to_string());
        let hidden_size = loaded.hidden_size();
        let vocab_size = loaded.vocab_size();
        let eos_token_ids = loaded.eos_token_ids().to_vec();
        let tokenizer = Arc::new(loaded.tokenizer().clone());
        let chat_template = Arc::new(loaded.chat_template().to_string());
        let arch = match &loaded {
            LoadedModel::Gemma(_) => LoadedArch::Gemma,
            LoadedModel::Qwen35(_) => LoadedArch::Qwen35,
            LoadedModel::Qwen3VlText(_) => LoadedArch::Qwen3VlText,
            LoadedModel::Deepseek4(_) => LoadedArch::Deepseek4,
        };

        // KV-spill descriptors: mirror Engine::spawn verbatim. The
        // SlotAware path inherits the same per-spill-family descriptor
        // construction; multi-slot spill semantics under SlotAware are
        // a future Phase A iter-5 concern (per ADR-040 §6 + risk R4-bis).
        let kv_spill_descriptor: Option<super::kv_spill_descriptor::KvSpillDescriptor> =
            match &loaded {
                LoadedModel::Gemma(g) => {
                    let kv_dtype = if crate::debug::INVESTIGATION_ENV.f16_kv {
                        super::kv_spill_descriptor::KvDType::F16
                    } else {
                        super::kv_spill_descriptor::KvDType::F32
                    };
                    let max_decode_tokens = 512usize;
                    let provenance = match &g.provenance {
                        crate::core::provenance::Provenance::Hf2q {
                            producer_version,
                            source_sha256,
                            ..
                        } => super::kv_spill_descriptor::KvSpillProvenance {
                            producer_version: producer_version.clone(),
                            source_sha256: source_sha256.clone(),
                            tokenizer_chat_template_hash:
                                super::kv_spill_descriptor::KvSpillProvenance::hash_chat_template(
                                    &g.chat_template,
                                ),
                        },
                        crate::core::provenance::Provenance::External => {
                            super::kv_spill_descriptor::KvSpillProvenance::default()
                        }
                    };
                    Some(
                        super::kv_spill_descriptor::KvSpillDescriptor::from_gemma_loaded_model(
                            &g.weights,
                            max_decode_tokens,
                            kv_dtype,
                            provenance,
                        ),
                    )
                }
                LoadedModel::Qwen35(_) => None,
                LoadedModel::Qwen3VlText(_) => None,
                LoadedModel::Deepseek4(_) => None,
            };

        let tq_packed_descriptor: Option<super::tq_packed_descriptor::TqPackedSpillDescriptor> =
            if super::tq_packed_descriptor::is_tq_active_mode() {
                match &loaded {
                    LoadedModel::Gemma(g) => {
                        let provenance_for_tq = match &g.provenance {
                            crate::core::provenance::Provenance::Hf2q {
                                producer_version,
                                source_sha256,
                                ..
                            } => super::kv_spill_descriptor::KvSpillProvenance {
                                producer_version: producer_version.clone(),
                                source_sha256: source_sha256.clone(),
                                tokenizer_chat_template_hash:
                                    super::kv_spill_descriptor::KvSpillProvenance::hash_chat_template(
                                        &g.chat_template,
                                    ),
                            },
                            crate::core::provenance::Provenance::External => {
                                super::kv_spill_descriptor::KvSpillProvenance::default()
                            }
                        };
                        super::tq_packed_descriptor::TqPackedSpillDescriptor::from_gemma_loaded_model_tq(
                            &g.weights,
                            provenance_for_tq,
                        )
                    }
                    LoadedModel::Qwen35(_)
                    | LoadedModel::Qwen3VlText(_)
                    | LoadedModel::Deepseek4(_) => None,
                }
            } else {
                None
            };

        let kv_spill_active = kv_spill_descriptor.is_some();
        let gguf = mlx_native::gguf::GgufFile::open(loaded.model_path())
            .expect("re-open loaded GGUF for SlotAware Engine load-info snapshot");
        let info = Arc::new(loaded.build_load_info(
            &gguf,
            loaded.load_duration(),
            kv_cache_budget_bytes,
            kv_spill_active,
        ));

        let registration = super::registry::find_for(&model_id);
        if let Some(ref r) = registration {
            tracing::info!(
                family = r.family,
                reasoning = r.has_reasoning(),
                tools = r.has_tools(),
                "hf2q-engine (SlotAware): matched model registration"
            );
        } else {
            tracing::info!(
                model_id = %model_id,
                "hf2q-engine (SlotAware): no matching model registration"
            );
        }

        let queue_capacity_u32 = queue_capacity.max(1) as u32;
        let initial_mode = EngineMode::SlotAware { max_slots };
        // Per ADR-040 §3.5: per-slot budget = total / max_slots. The
        // SlotAware path is the FIRST callsite where this division
        // matters (SerialFifo divides by 1).
        let initial_per_slot_kv_budget_bytes: u64 = kv_cache_budget_bytes
            .map(|b| b / u64::from(max_slots.max(1)))
            .unwrap_or(0);
        let initial_kv_bytes_per_token_cached: u64 = info.kv_bytes_per_token();

        // Seed SchedulerStats with the InflightBatched policy so a
        // pre-admit /metrics scrape reports the configured shape.
        let scheduler_stats_snapshot = Arc::new(Mutex::new(SchedulerStats {
            policy: SchedulerPolicy::InflightBatched,
            in_flight_slots: 0,
            queue_capacity: queue_capacity_u32,
            admitted_total: 0,
            rejected_429_total: 0,
            completed_total: 0,
        }));
        let worker_stats_handle = Arc::clone(&scheduler_stats_snapshot);

        let worker_registration = registration.clone();
        let worker_per_slot_budget = initial_per_slot_kv_budget_bytes;
        let worker_kv_bytes_per_token = initial_kv_bytes_per_token_cached;
        let worker_handle = std::thread::Builder::new()
            .name("hf2q-engine-slotaware".into())
            .spawn(move || {
                worker_run(
                    loaded,
                    rx,
                    worker_registration,
                    initial_mode,
                    queue_capacity_u32,
                    worker_stats_handle,
                    worker_per_slot_budget,
                    worker_kv_bytes_per_token,
                )
            })
            .expect("spawn hf2q-engine-slotaware thread");

        Engine {
            inner: Arc::new(EngineInner {
                tx,
                worker_handle: Mutex::new(Some(worker_handle)),
                info,
                arch,
                model_id,
                context_length,
                quant_type,
                hidden_size,
                vocab_size,
                eos_token_ids,
                tokenizer,
                chat_template,
                registration,
                token_bytes: std::sync::OnceLock::new(),
                kv_spill_descriptor,
                tq_packed_descriptor,
                mode: initial_mode,
                max_slots,
                per_slot_kv_budget_bytes: initial_per_slot_kv_budget_bytes,
                kv_bytes_per_token_cached: initial_kv_bytes_per_token_cached,
                scheduler_stats_snapshot,
            }),
        }
    }

    /// ADR-040 Phase C iter-1.5 — read back the engine's mode.
    ///
    /// Returns the [`EngineMode`] stored on [`EngineInner`] at spawn time
    /// — for engines built via the 3-arg [`Engine::spawn`] this is always
    /// [`EngineMode::SerialFifo`]; for engines built via
    /// [`Engine::spawn_with_mode`] this echoes the validated requested
    /// mode.
    ///
    /// Per ADR-040 §7 + the iter-1.5 adversarial review, this accessor
    /// MUST NOT lie about the requested mode (iter-1's "always return
    /// `EngineMode::default()`" was a Liskov-substitution violation).
    pub fn mode(&self) -> EngineMode {
        self.inner.mode
    }

    /// **ADR-040 Phase C iter-2a (C2b)** — slot cap snapshot.
    ///
    /// Always `1` under `EngineMode::SerialFifo` (the only mode that
    /// survives validation at iter-2a); `max_slots` from `SlotAware`
    /// once iter-2b lifts the [`EngineSpawnError::ModeNotYetWired`]
    /// rejection.
    pub fn max_slots(&self) -> u32 {
        self.inner.max_slots
    }

    /// **ADR-040 §3.5 iter-A5b** — per-slot KV byte budget configured
    /// at spawn time (`kv_cache_budget_bytes / max_slots`).
    ///
    /// `0` means enforcement is disabled (operator did not set
    /// `--kv-cache-budget-bytes`, or the synthetic-fixture
    /// loader path). Exposed for handler-side pre-stream admit checks
    /// + Prometheus exposition.
    pub fn per_slot_kv_budget_bytes(&self) -> u64 {
        self.inner.per_slot_kv_budget_bytes
    }

    /// **ADR-040 §3.5 iter-A5b** — pre-stream admit-time KV byte
    /// budget check. Returns `Ok(())` when the request fits the
    /// per-slot budget (or enforcement is disabled), or
    /// [`EngineAdmitError::SlotBudgetExceeded`] when the projected
    /// KV cost exceeds the per-slot budget configured at engine spawn.
    ///
    /// **Why pre-stream**: per codex review CRITICAL #2 (handlers.rs:
    /// 1739-1748 originally string-matched `queue_full` only),
    /// scheduler-side `SlotBudgetExceeded` rejection in the streaming
    /// arm landed AFTER `Engine::generate_stream_with_deepstack`
    /// returned `Ok`, meaning the handler had already committed to
    /// opening an SSE body. Calling this method BEFORE handing the
    /// request to `generate_stream_with_deepstack` lets the streaming
    /// handler return a clean HTTP 429 + `Retry-After: 1` body
    /// instead of a half-rendered SSE error frame.
    ///
    /// Defense-in-depth: the worker_run admit sites still surface
    /// `slot_budget_exceeded`-prefixed anyhow errors for the
    /// non-streaming path; the handler-side string-match converts
    /// those to `ApiError::slot_budget_exceeded` parallel to
    /// `queue_full`. Pre-stream check + worker-side check are
    /// redundant by design — both surfaces map to the same wire-level
    /// 429.
    ///
    /// Behaviour:
    /// - `per_slot_kv_budget_bytes == 0` ⇒ `Ok(())` (enforcement
    ///   disabled; preserves pre-A5 byte-equivalence for operators
    ///   who did not set `--kv-cache-budget-bytes`).
    /// - `kv_bytes_per_token == 0` ⇒ `Ok(())` (synthetic fixture /
    ///   LoadInfo arch facts missing — treat as "do not enforce" to
    ///   match the scheduler-side opt-out contract).
    /// - Otherwise: compute
    ///   `(prompt_tokens + max_tokens) × kv_bytes_per_token` and
    ///   return `Err(SlotBudgetExceeded { .. })` if it exceeds the
    ///   per-slot budget.
    pub fn try_admit_budget(
        &self,
        prompt_tokens: u32,
        max_tokens: u32,
    ) -> std::result::Result<(), EngineAdmitError> {
        let budget = self.inner.per_slot_kv_budget_bytes;
        let per_token = self.inner.kv_bytes_per_token_cached;
        if budget == 0 || per_token == 0 {
            // Enforcement disabled — preserves pre-A5
            // byte-equivalence verbatim.
            return Ok(());
        }
        let needed = u64::from(prompt_tokens)
            .saturating_add(u64::from(max_tokens))
            .saturating_mul(per_token);
        if needed > budget {
            return Err(EngineAdmitError::SlotBudgetExceeded {
                needed_bytes: needed,
                budget_bytes: budget,
            });
        }
        Ok(())
    }

    /// **ADR-040 Phase C iter-2a (C2b)** — most recent
    /// [`SchedulerStats`] snapshot written by the worker thread.
    ///
    /// The worker writes a snapshot after each `release` (FIFO
    /// completion); handlers read for `/metrics` (Phase C3 wiring).
    /// On a freshly-spawned engine that has not yet processed any
    /// requests, the snapshot mirrors the configured `queue_capacity`
    /// with zero counters.
    ///
    /// Returns a cloned [`SchedulerStats`] — the lock is held only for
    /// the duration of the clone (a 32-byte memcpy plus a discriminant).
    pub fn scheduler_stats(&self) -> SchedulerStats {
        self.inner
            .scheduler_stats_snapshot
            .lock()
            .expect("ADR-040 C2b: scheduler_stats_snapshot mutex poisoned")
            .clone()
    }

    /// Iter-215 Wedge-2: which `LoadedModel` variant the worker
    /// thread owns.  Used by handlers (chat, embeddings, vision) to
    /// short-circuit to HTTP 501 when the variant's inference path
    /// is not yet implemented (today: `LoadedArch::Qwen35`).
    pub fn arch(&self) -> LoadedArch {
        self.inner.arch
    }

    /// Unified load snapshot for this engine.
    pub fn info(&self) -> &LoadInfo {
        &self.inner.info
    }

    /// Lazily build + cache the per-vocab decoded UTF-8 byte table used
    /// by the grammar mask (Phase 2a Task #5 / iter-95).
    ///
    /// `token_bytes[id]` is the bytes the tokenizer emits when token `id`
    /// is sampled — exactly `tokenizer.decode(&[id], false)` lowered to
    /// raw UTF-8 bytes.  Empty entries (special / unprintable tokens
    /// like `<eos>` / `<turn|>`) are left blank; the mask treats them
    /// as "do not constrain" — the decode loop's EOS/stop-string layer
    /// owns those.
    ///
    /// Cost: vocab × one tokenizer.decode call.  At Gemma-4's vocab=256K
    /// this is ~50-200 ms on first call (CPU work; not on hot path),
    /// then ~free for every subsequent grammar request through this
    /// Engine.  The build runs on the calling thread so the worker
    /// thread is unaffected.
    ///
    /// Returned as `Arc<Vec<Vec<u8>>>` — the chat handler attaches it
    /// to `SamplingParams.token_bytes` (cheap Arc clone) so the worker
    /// thread can consume it without re-resolving on every request.
    pub fn token_bytes_table(&self) -> Arc<Vec<Vec<u8>>> {
        self.inner
            .token_bytes
            .get_or_init(|| {
                let v = self.inner.vocab_size;
                let tok = &self.inner.tokenizer;
                let mut out: Vec<Vec<u8>> = Vec::with_capacity(v);
                for id in 0..v as u32 {
                    // `decode` returns the rendered text per token; for
                    // BPE-byte-fallback vocabs the bytes round-trip via
                    // UTF-8.  Failure (out-of-range id, unsupported
                    // token) returns an empty string — emit empty bytes
                    // so the mask treats it as a "special" / unprintable
                    // token (left untouched).
                    let s = tok.decode(&[id], false).unwrap_or_default();
                    out.push(s.into_bytes());
                }
                tracing::info!(
                    "Engine: built per-vocab token_bytes table ({} ids, ~{:.1} MB)",
                    v,
                    out.iter().map(|v| v.len()).sum::<usize>() as f64 / 1e6
                );
                Arc::new(out)
            })
            .clone()
    }

    pub fn model_id(&self) -> &str {
        &self.inner.model_id
    }
    pub fn context_length(&self) -> Option<usize> {
        self.inner.context_length
    }
    pub fn quant_type(&self) -> Option<&str> {
        self.inner.quant_type.as_deref()
    }
    /// Hidden-state dimensionality.  Used by the `/v1/embeddings` handler
    /// when the chat model is the embedder (Phase 2a Task #8).
    pub fn hidden_size(&self) -> usize {
        self.inner.hidden_size
    }
    pub fn tokenizer(&self) -> &Tokenizer {
        &self.inner.tokenizer
    }
    pub fn chat_template(&self) -> &str {
        &self.inner.chat_template
    }
    pub fn eos_token_ids(&self) -> &[u32] {
        &self.inner.eos_token_ids
    }
    pub fn registration(&self) -> Option<&super::registry::ModelRegistration> {
        self.inner.registration.as_ref()
    }

    /// Phase B-dense.2 follow-up: cached KV-spill shape descriptor.
    /// Populated for the `Gemma` variant at spawn time; `None` for the
    /// `Qwen35` variant (its KV state is hybrid and belongs to a future
    /// B-hybrid descriptor).
    ///
    /// Read-only reference into the engine's interior; no lock taken,
    /// no worker round-trip — descriptor was captured synchronously
    /// from the `MlxModelWeights` before the move into the worker
    /// thread. Used by
    /// `Gemma4DenseSpillFactory::try_from_engine_arc` to construct a
    /// real (non-stub) hook from the live shape.
    pub fn kv_spill_descriptor(&self) -> Option<&super::kv_spill_descriptor::KvSpillDescriptor> {
        self.inner.kv_spill_descriptor.as_ref()
    }

    /// **ADR-017 §B-tq.4 iter-4** — accessor for the per-layer
    /// TQ-packed runtime shape.  `None` unless `HF2Q_TQ_KV=1` AND
    /// loaded model is Gemma 4.  Read by
    /// `TqPackedSpillFactory::try_construct` to build a per-layer-
    /// correct cfg.
    pub fn tq_packed_descriptor(
        &self,
    ) -> Option<&super::tq_packed_descriptor::TqPackedSpillDescriptor> {
        self.inner.tq_packed_descriptor.as_ref()
    }

    /// Phase B-dense.2 follow-up: synchronously read a layer's
    /// `dense_kvs[layer_rank]` K/V byte slice over the given token-
    /// position `range`. Sends a `Request::KvSnapshot` to the worker
    /// thread and blocks until the reply arrives.
    ///
    /// Returns:
    /// - `Ok(Some(KvSnapshotBytes))` on Gemma variant with populated
    ///   `dense_kvs` (post-prefill).
    /// - `Ok(None)` on Gemma variant with `dense_kvs == None` (no
    ///   prefill yet) OR on Qwen35 variant (no dense KV).
    /// - `Err(...)` on worker channel closure or layer-out-of-range.
    ///
    /// ## Synchronous semantics
    ///
    /// The hook calls this from
    /// `Gemma4DenseSpill::snapshot_block(&self, ...)` which is `&self`
    /// (no async). We use a tokio runtime handle if available — the
    /// hook runs from `HotSwapManager::evict` / `load_or_get` which
    /// are async paths (per multi_model.rs:887-892). Falls back to a
    /// `blocking_send` + `blocking_recv` pair when called from a
    /// non-tokio context (e.g. unit tests on the main thread).
    /// ADR-017 Closure iter-5 / Phase E (2026-05-04) — synchronously
    /// snapshot the loaded model's `PromptCache` into a JSON byte
    /// payload. Returns `Ok(None)` when the cache is empty or
    /// grammar-bound (see [`crate::serve::kv_persist::prompt_cache_persist`]
    /// module docs).
    ///
    /// Drives a `Request::PromptCacheSnapshot` round-trip through the
    /// worker thread (sole owner of `loaded.prompt_cache`). Same
    /// channel + blocking-send/recv pattern as `request_kv_snapshot`.
    pub fn request_prompt_cache_snapshot(&self) -> Result<Option<Vec<u8>>> {
        let (reply_tx, reply_rx) = oneshot::channel();
        let req = Request::PromptCacheSnapshot { reply: reply_tx };
        match self.inner.tx.try_send(req) {
            Ok(()) => {}
            Err(mpsc::error::TrySendError::Full(req)) => {
                self.inner
                    .tx
                    .blocking_send(req)
                    .context("engine worker is gone (prompt_cache_snapshot full→blocking)")?;
            }
            Err(mpsc::error::TrySendError::Closed(_)) => {
                anyhow::bail!("engine worker is gone (prompt_cache_snapshot)");
            }
        }
        reply_rx
            .blocking_recv()
            .context("prompt_cache_snapshot reply dropped")?
    }

    /// ADR-017 Closure iter-5 / Phase E (2026-05-04) — synchronously
    /// restore the loaded model's `PromptCache` from a JSON byte
    /// payload. Returns `Err(...)` on parse failure / version
    /// mismatch / non-Gemma model.
    pub fn request_prompt_cache_restore(&self, payload: Vec<u8>) -> Result<()> {
        let (reply_tx, reply_rx) = oneshot::channel();
        let req = Request::PromptCacheRestore {
            payload,
            reply: reply_tx,
        };
        match self.inner.tx.try_send(req) {
            Ok(()) => {}
            Err(mpsc::error::TrySendError::Full(req)) => {
                self.inner
                    .tx
                    .blocking_send(req)
                    .context("engine worker is gone (prompt_cache_restore full→blocking)")?;
            }
            Err(mpsc::error::TrySendError::Closed(_)) => {
                anyhow::bail!("engine worker is gone (prompt_cache_restore)");
            }
        }
        reply_rx
            .blocking_recv()
            .context("prompt_cache_restore reply dropped")?
    }

    pub fn request_kv_snapshot(
        &self,
        layer_rank: usize,
        range: std::ops::Range<u32>,
    ) -> Result<Option<KvSnapshotBytes>> {
        let (reply_tx, reply_rx) = oneshot::channel();
        let req = Request::KvSnapshot {
            layer_rank,
            range,
            reply: reply_tx,
        };
        // Send synchronously via the tokio Sender's `blocking_send`
        // helper. Works from any thread; if called from inside a
        // tokio runtime task, prefer `Handle::block_on(tx.send(...))`
        // path to avoid the "blocking inside async" warning. The
        // `try_send` path is non-blocking and surfaces queue-full
        // immediately; we fall back to `blocking_send` if try_send
        // returns full because KV snapshot is a control-plane request
        // that should NOT silently fail.
        match self.inner.tx.try_send(req) {
            Ok(()) => {}
            Err(mpsc::error::TrySendError::Full(req)) => {
                // Queue full — do a blocking send so the request is
                // eventually delivered. Acceptable because KV snapshot
                // is rare (eviction-time only) and the FIFO already
                // forces serialization.
                self.inner
                    .tx
                    .blocking_send(req)
                    .context("engine worker is gone (kv_snapshot full→blocking)")?;
            }
            Err(mpsc::error::TrySendError::Closed(_)) => {
                anyhow::bail!("engine worker is gone (kv_snapshot)");
            }
        }
        reply_rx
            .blocking_recv()
            .context("kv_snapshot reply dropped")?
    }

    /// Phase B-dense.2 follow-up: synchronously write a layer's
    /// `dense_kvs[layer_rank]` K/V byte slice over the given token-
    /// position `range`. Sends a `Request::KvRestore` to the worker
    /// thread and blocks until the reply arrives.
    ///
    /// `k_payload` and `v_payload` are head-major raw bytes
    /// (`[nkv_heads, n_tokens, head_dim]`); the worker copies them
    /// directly into the live `MlxBuffer` via `as_mut_slice::<u8>()`
    /// at the slot positions implied by `range` (ring-buffer for
    /// sliding layers, linear for full-attention).
    ///
    /// `write_pos` is the sliding-ring write position to restore
    /// (`u32::MAX` sentinel for full-attention layers).
    ///
    /// Returns `Err(...)` on worker channel closure, layer-out-of-
    /// range, shape mismatch, or allocation failure.
    pub fn request_kv_restore(
        &self,
        layer_rank: usize,
        range: std::ops::Range<u32>,
        k_payload: Vec<u8>,
        v_payload: Vec<u8>,
        write_pos: u32,
    ) -> Result<()> {
        let (reply_tx, reply_rx) = oneshot::channel();
        let req = Request::KvRestore {
            layer_rank,
            range,
            k_payload,
            v_payload,
            write_pos,
            reply: reply_tx,
        };
        match self.inner.tx.try_send(req) {
            Ok(()) => {}
            Err(mpsc::error::TrySendError::Full(req)) => {
                self.inner
                    .tx
                    .blocking_send(req)
                    .context("engine worker is gone (kv_restore full→blocking)")?;
            }
            Err(mpsc::error::TrySendError::Closed(_)) => {
                anyhow::bail!("engine worker is gone (kv_restore)");
            }
        }
        reply_rx
            .blocking_recv()
            .context("kv_restore reply dropped")?
    }

    /// **Phase B-tq.4** — synchronously snapshot a layer's TQ-packed
    /// K/V state for the given token-position `range`.  Sends a
    /// `Request::TqPackedKvSnapshot` to the worker thread; worker
    /// reads `MlxModelWeights.kv_caches[layer]` and packs two
    /// `tq_packed_v2` envelopes (one for K, one for V).  Returns
    /// `(k_payload, v_payload)`.
    ///
    /// Mirror of [`Self::request_kv_snapshot`] for the TurboQuant-
    /// active KV path.  Called by
    /// [`crate::serve::kv_persist::families::tq_packed::TqPackedSpill::snapshot_via_engine`]
    /// from inside `KvCacheSpill::snapshot_block`.
    pub fn tq_packed_v2_snapshot_block(
        &self,
        layer_rank: usize,
        range: std::ops::Range<u32>,
        bits_per_coord: crate::serve::kv_persist::families::tq_packed::TqBitsPerCoord,
        flags: u32,
        scale: f64,
    ) -> Result<(Vec<u8>, Vec<u8>)> {
        let (reply_tx, reply_rx) = oneshot::channel();
        let req = Request::TqPackedKvSnapshot {
            layer_rank,
            range,
            bits_per_coord,
            flags,
            scale,
            reply: reply_tx,
        };
        match self.inner.tx.try_send(req) {
            Ok(()) => {}
            Err(mpsc::error::TrySendError::Full(req)) => {
                self.inner
                    .tx
                    .blocking_send(req)
                    .context("engine worker is gone (tq_packed_kv_snapshot full→blocking)")?;
            }
            Err(mpsc::error::TrySendError::Closed(_)) => {
                anyhow::bail!("engine worker is gone (tq_packed_kv_snapshot)");
            }
        }
        reply_rx
            .blocking_recv()
            .context("tq_packed_kv_snapshot reply dropped")?
    }

    /// **Phase B-tq.4** — synchronously restore a layer's TQ-packed
    /// K/V state from `(k_payload, v_payload)` envelopes.  Inverse of
    /// [`Self::tq_packed_v2_snapshot_block`].
    pub fn tq_packed_v2_restore_block(
        &self,
        layer_rank: usize,
        range: std::ops::Range<u32>,
        bits_per_coord: crate::serve::kv_persist::families::tq_packed::TqBitsPerCoord,
        k_payload: &[u8],
        v_payload: &[u8],
    ) -> Result<()> {
        let (reply_tx, reply_rx) = oneshot::channel();
        let req = Request::TqPackedKvRestore {
            layer_rank,
            range,
            bits_per_coord,
            k_payload: k_payload.to_vec(),
            v_payload: v_payload.to_vec(),
            reply: reply_tx,
        };
        match self.inner.tx.try_send(req) {
            Ok(()) => {}
            Err(mpsc::error::TrySendError::Full(req)) => {
                self.inner
                    .tx
                    .blocking_send(req)
                    .context("engine worker is gone (tq_packed_kv_restore full→blocking)")?;
            }
            Err(mpsc::error::TrySendError::Closed(_)) => {
                anyhow::bail!("engine worker is gone (tq_packed_kv_restore)");
            }
        }
        reply_rx
            .blocking_recv()
            .context("tq_packed_kv_restore reply dropped")?
    }

    /// Run a single-prompt warmup pass. Blocks until the worker finishes it.
    /// Typical cost is one prefill + a few decode tokens on a tiny prompt —
    /// at the 10ms-order on M5 Max. The warmup's job is to compile all
    /// kernels and fault in hot weights so the first real request doesn't
    /// pay the one-time setup latency.
    pub async fn warmup(&self) -> Result<()> {
        let (reply_tx, reply_rx) = oneshot::channel();
        self.inner
            .tx
            .send(Request::Warmup { reply: reply_tx })
            .await
            .context("engine worker is gone")?;
        reply_rx.await.context("warmup reply dropped")?
    }

    /// Enqueue a non-streaming generation. Returns `queue_full` if the FIFO
    /// is at capacity (handlers map to 429 + Retry-After).
    pub async fn generate(
        &self,
        prompt_tokens: Vec<u32>,
        params: SamplingParams,
    ) -> Result<GenerationResult> {
        let (reply_tx, reply_rx) = oneshot::channel();
        let req = Request::Generate {
            prompt_tokens,
            params,
            reply: reply_tx,
        };
        // Use `try_send` so we can distinguish queue-full from a closed worker.
        match self.inner.tx.try_send(req) {
            Ok(()) => {}
            Err(mpsc::error::TrySendError::Full(_)) => {
                anyhow::bail!("queue_full");
            }
            Err(mpsc::error::TrySendError::Closed(_)) => {
                anyhow::bail!("engine worker is gone");
            }
        }
        reply_rx.await.context("generation reply dropped")?
    }

    /// Enqueue a streaming generation. The caller owns `events_rx` (returned
    /// separately) and wraps it in the SSE encoder. Returns immediately after
    /// queueing; the worker emits tokens into `events_tx` as they decode.
    ///
    /// Dropping `events_rx` (handler-side) causes the next worker `send` to
    /// fail and the worker aborts the decode loop, freeing the queue slot
    /// (Decision #18). Queue-full returns an error that the handler maps to
    /// 429 + Retry-After.
    pub async fn generate_stream(
        &self,
        prompt_tokens: Vec<u32>,
        params: SamplingParams,
        events_tx: mpsc::Sender<super::sse::GenerationEvent>,
        cancellation_counter: Option<Arc<std::sync::atomic::AtomicU64>>,
        soft_tokens: Vec<SoftTokenData>,
    ) -> Result<()> {
        self.generate_stream_with_deepstack(
            prompt_tokens,
            params,
            events_tx,
            cancellation_counter,
            soft_tokens,
            None,
            None,
        )
        .await
    }

    /// **Wedge-4e (iter-224 row 5)**: streaming-with-soft-tokens entry
    /// extended with optional `deepstack: Option<DeepstackData>` and
    /// `positions_flat: Option<Vec<i32>>` for the Qwen3-VL streaming
    /// path. When both are `None`, behaviour is byte-identical to the
    /// legacy `generate_stream` (which still exists as a thin wrapper
    /// passing `None` for both).
    ///
    /// This is the single seam through which the chat handler routes
    /// streaming Qwen3-VL chat (image-bearing + tools[] + reasoning)
    /// — the worker thread rebuilds borrowed `DeepstackInjection<'_>`
    /// slices and dispatches through the
    /// `forward_gpu_last_logits_with_soft_tokens_and_deepstack` LM
    /// forward, mirroring the non-streaming
    /// `generate_with_soft_tokens_and_deepstack` shape.
    ///
    /// The MODE-INVARIANT `ReasoningSplitter` and `ToolCallSplitter`
    /// chain (Wedge-3 Phase E) sees the per-token decoded fragments
    /// regardless of prefill source, so reasoning_content + tool_calls
    /// surface end-to-end on multimodal streaming requests.
    pub async fn generate_stream_with_deepstack(
        &self,
        prompt_tokens: Vec<u32>,
        params: SamplingParams,
        events_tx: mpsc::Sender<super::sse::GenerationEvent>,
        cancellation_counter: Option<Arc<std::sync::atomic::AtomicU64>>,
        soft_tokens: Vec<SoftTokenData>,
        deepstack: Option<DeepstackData>,
        positions_flat: Option<Vec<i32>>,
    ) -> Result<()> {
        let req = Request::GenerateStream {
            prompt_tokens,
            params,
            events: events_tx,
            cancellation_counter,
            soft_tokens,
            deepstack,
            positions_flat,
        };
        match self.inner.tx.try_send(req) {
            Ok(()) => Ok(()),
            Err(mpsc::error::TrySendError::Full(_)) => anyhow::bail!("queue_full"),
            Err(mpsc::error::TrySendError::Closed(_)) => anyhow::bail!("engine worker is gone"),
        }
    }

    /// Enqueue a pooled-embedding request (ADR-005 Task #8, iter-92).
    ///
    /// Returns the L2-normalized last-token hidden state as a `Vec<f32>`
    /// of length `hidden_size`.  Uses the same FIFO queue + 429 semantics
    /// as `generate`: a full queue maps to `queue_full` (handler maps to
    /// HTTP 429 + Retry-After).
    pub async fn embed(&self, prompt_tokens: Vec<u32>) -> Result<Vec<f32>> {
        let (reply_tx, reply_rx) = oneshot::channel();
        let req = Request::Embed {
            prompt_tokens,
            reply: reply_tx,
        };
        match self.inner.tx.try_send(req) {
            Ok(()) => {}
            Err(mpsc::error::TrySendError::Full(_)) => {
                anyhow::bail!("queue_full");
            }
            Err(mpsc::error::TrySendError::Closed(_)) => {
                anyhow::bail!("engine worker is gone");
            }
        }
        reply_rx.await.context("embedding reply dropped")?
    }

    /// Vision-aware non-streaming generation (Phase 2c Task #17 / iter-98).
    ///
    /// Same as `generate` but passes per-position embedding overrides
    /// that the worker plugs into the prefill via
    /// `MlxModelWeights::forward_prefill_with_soft_tokens`.  Used by
    /// the chat handler when an `image_url` content part is present
    /// in the request: the projected vision embeddings for each image
    /// flow through this API as `SoftTokenData` covering the
    /// placeholder-token positions in the prompt.
    pub async fn generate_with_soft_tokens(
        &self,
        prompt_tokens: Vec<u32>,
        soft_tokens: Vec<SoftTokenData>,
        params: SamplingParams,
    ) -> Result<GenerationResult> {
        self.generate_with_soft_tokens_and_deepstack(prompt_tokens, soft_tokens, params, None, None)
            .await
    }

    /// Wedge-4d entry point for Qwen3-VL chat: same as
    /// `generate_with_soft_tokens` but threads a `DeepstackData` (one
    /// chunk per ViT-flagged-layer head) and a 3D-mRoPE
    /// `positions_flat` buffer through to the worker. Both
    /// `deepstack` and `positions_flat` are `None` for Gemma /
    /// non-Qwen3-VL paths, in which case behaviour is byte-identical
    /// to the legacy `generate_with_soft_tokens` entry point.
    pub async fn generate_with_soft_tokens_and_deepstack(
        &self,
        prompt_tokens: Vec<u32>,
        soft_tokens: Vec<SoftTokenData>,
        params: SamplingParams,
        deepstack: Option<DeepstackData>,
        positions_flat: Option<Vec<i32>>,
    ) -> Result<GenerationResult> {
        let (reply_tx, reply_rx) = oneshot::channel();
        let req = Request::GenerateWithSoftTokens {
            prompt_tokens,
            soft_tokens,
            params,
            deepstack,
            positions_flat,
            reply: reply_tx,
        };
        match self.inner.tx.try_send(req) {
            Ok(()) => {}
            Err(mpsc::error::TrySendError::Full(_)) => {
                anyhow::bail!("queue_full");
            }
            Err(mpsc::error::TrySendError::Closed(_)) => {
                anyhow::bail!("engine worker is gone");
            }
        }
        reply_rx.await.context("vision generation reply dropped")?
    }

    /// Request a clean shutdown of the worker. Drains in-flight + queued work
    /// (FIFO ordering means the `Shutdown` sentinel runs after every request
    /// already enqueued) and then joins the worker thread.
    ///
    /// Returns `Ok(())` once the worker thread has fully exited; returns
    /// `Err(...)` only if the join itself panicked.
    ///
    /// Idempotent: calling twice (or on a clone whose sibling already
    /// joined) is a no-op — the second call observes `worker_handle = None`
    /// and returns immediately. The blocking `.join()` runs inside
    /// `tokio::task::spawn_blocking` so the calling tokio runtime is not
    /// blocked while the worker drains a long generation.
    pub async fn shutdown(&self) -> Result<()> {
        // Send Shutdown sentinel. Errors here mean the worker tx was already
        // dropped/closed — the thread has already exited; treat as success
        // for the join step below.
        let _ = self.inner.tx.send(Request::Shutdown).await;

        // Take the JoinHandle exactly once. Subsequent shutdown() calls
        // observe None and return Ok(()).
        let handle = match self.inner.worker_handle.lock() {
            Ok(mut guard) => guard.take(),
            Err(_) => None, // poisoned mutex => worker already gone
        };

        if let Some(handle) = handle {
            // Join can block until the in-flight Generate finishes; do it
            // off-runtime so we don't stall axum's drain phase. The handle's
            // ownership has already been moved out of `inner`, so this
            // closure can take it.
            tokio::task::spawn_blocking(move || handle.join())
                .await
                .context("spawn_blocking for worker join")?
                .map_err(|panic| {
                    anyhow::anyhow!("engine worker thread panicked on shutdown: {:?}", panic)
                })?;
        }

        Ok(())
    }
}

// ---------------------------------------------------------------------------
// Iter-215 Wedge-2 — Qwen3.5/3.6 SERVE-side 501 sentinel
// ---------------------------------------------------------------------------
//
// The worker thread for `LoadedModel::Qwen35` returns a sentinel error
// for every inference request in iter-215 MVP.  The chat-completion
// handler matches on the sentinel substring and maps it to HTTP 501.
// Wedge-3 (deferred follow-up) replaces this arm with the real
// `Qwen35Model::forward_*` pipeline.
//
// The message MUST contain BOTH `hf2q generate` AND `cmd_generate_qwen35`
// literals — operator-actionable contract verified by the iter-215
// tests in this file and `serve/api/router.rs`.

/// Sentinel substring the chat / embedding / vision handlers match on
/// to map worker errors to HTTP 501 (Not Implemented).  Iter-215 MVP
/// only — Wedge-3 removes this once forward_gpu lands.
pub const QWEN35_NOT_IMPLEMENTED_SENTINEL: &str = "qwen35_not_implemented";

/// Operator-facing message body emitted on the 501 path.  Names both
/// the working CLI alternative (`hf2q generate`) AND the function that
/// implements it (`cmd_generate_qwen35`) so an operator can grep the
/// codebase and confirm the surface is real, not a stub.
pub const QWEN35_NOT_IMPLEMENTED_MESSAGE: &str =
    "Qwen3.5/3.6 chat completion via the SERVE-side path is pending Phase E (Wedge-3). \
     The model is loaded; /readyz, /v1/models, /metrics work. For chat completions today, \
     use `hf2q generate --model <path> --prompt <text>` which routes correctly via \
     cmd_generate_qwen35.";

/// Build the anyhow::Error the worker sends back when a chat / embed /
/// vision request lands on a Qwen35 variant.  The error message
/// contains the sentinel + the operator-facing message.  The chat
/// handler matches on `QWEN35_NOT_IMPLEMENTED_SENTINEL` to dispatch
/// to a 501 response.
pub(crate) fn qwen35_not_implemented_err<T>() -> Result<T> {
    Err(anyhow::anyhow!(
        "{}: {}",
        QWEN35_NOT_IMPLEMENTED_SENTINEL,
        QWEN35_NOT_IMPLEMENTED_MESSAGE
    ))
}

/// Worker-thread entry point. Owns the `LoadedModel` and drains requests
/// serially. `registration` (if `Some`) drives reasoning-content split
/// (Decision #21) — decode text passes through a `ReasoningSplitter` on
/// the way out.
///
/// Iter-215 Wedge-2: dispatches on the `LoadedModel` enum variant.
/// `LoadedModel::Gemma` runs the production `forward_mlx` path
/// unchanged; `LoadedModel::Qwen35` returns the iter-215 MVP 501
/// sentinel error (`QWEN35_NOT_IMPLEMENTED_SENTINEL`) which the chat
/// handler maps to HTTP 501 with an operator-actionable message.
/// Wedge-3 (deferred follow-up) replaces the 501 arm with the actual
/// `Qwen35Model::forward_*` chain.
/// **ADR-040 Phase C iter-2c (C2c)** — enum dispatcher over the two
/// concrete scheduler variants the worker thread owns.
///
/// Per dossier §2.9 the `Scheduler` trait surface deliberately omits
/// `advance_after_prefill` / `advance_after_decode` (their FSM-advance
/// shapes differ between FIFO and InflightBatched). The C2b worker held
/// a concrete `FifoSchedulerAdapter`; iter-C2c adds the `Inflight` arm
/// for `EngineMode::SlotAware`. This enum gives the worker uniform
/// access to `admit` / `release` / `stats` / `advance_after_*` without
/// `Box<dyn Scheduler>` (which would lose access to the
/// type-specific advance APIs).
///
/// **Path B scope** (iter-C2c): under SlotAware the worker still
/// dispatches one request at a time (Shape A R2 limitation per dossier
/// §2.7); the InflightBatched scheduler hands out `SlotId(0)` for
/// every admit (the free list recycles slot 0 on every release). At
/// admit time `slot_id > 0` returns
/// `MultiSeqError::CapabilityUnsupported` via the worker arm (typed
/// deferral for iter-C2c-cont kernel slot routing). Concurrent
/// admission of N>1 requests requires Shape B's `tokio::select!` body
/// (iter-C2c-cont per ADR-040 §6 + dossier §2.7).
enum WorkerScheduler {
    Fifo(FifoSchedulerAdapter),
    Inflight(InflightBatchedScheduler),
}

impl WorkerScheduler {
    fn admit(
        &mut self,
        req: AdmitRequest,
    ) -> Result<crate::serve::scheduler::RequestSlot, AdmitError> {
        match self {
            Self::Fifo(s) => s.admit(req),
            Self::Inflight(s) => s.admit(req),
        }
    }

    fn release(&mut self, handle: SlotHandle) {
        match self {
            Self::Fifo(s) => s.release(handle),
            Self::Inflight(s) => s.release(handle),
        }
    }

    fn advance_after_prefill(&mut self, handle: SlotHandle, n_consumed: u32) {
        match self {
            Self::Fifo(s) => s.advance_after_prefill(handle, n_consumed),
            Self::Inflight(s) => s.advance_after_prefill(handle, n_consumed),
        }
    }

    fn advance_after_decode(&mut self, handle: SlotHandle) {
        match self {
            Self::Fifo(s) => s.advance_after_decode(handle),
            Self::Inflight(s) => s.advance_after_decode(handle),
        }
    }

    /// ADR-040 Phase F M1 (F1) — drive the scheduler one tick.
    ///
    /// This is the FIRST callsite of `Scheduler::step()` in the worker:
    /// the Phase A–E scaffold built `InflightBatchedScheduler::step`
    /// (scheduler.rs:1266) correctly — promote one queued slot, pick the
    /// oldest `Prefilling` slot (FIFO), gather every `Decoding` handle —
    /// but nothing ever called it (the SerialFifo drain at the
    /// `blocking_recv` loop ran each request's whole generate loop
    /// inline). `worker_run_slot_aware` calls this every tick to obtain
    /// the next `SchedulerStep` (Idle / Prefill / Decode / Mixed). The
    /// FIFO arm delegates to the trait `step()` too so the enum stays a
    /// faithful pass-through; SerialFifo never reaches this method
    /// because it runs the legacy inline drain.
    fn step(&mut self) -> Result<SchedulerStep, StepError> {
        match self {
            Self::Fifo(s) => Scheduler::step(s),
            Self::Inflight(s) => Scheduler::step(s),
        }
    }

    fn stats(&self) -> SchedulerStats {
        match self {
            Self::Fifo(s) => s.stats(),
            Self::Inflight(s) => s.stats(),
        }
    }
}

// ===========================================================================
// ADR-040 Phase F M1 (F1) — scheduler-driven, admit-while-decoding worker
// loop for `EngineMode::SlotAware`.
//
// The Phase A–E scaffold built the multi-seq KV substrate, the
// `InflightBatchedScheduler` (with a correct batched `step()`), and the
// per-arch slot-aware generate functions — but never DROVE the scheduler:
// the worker drained one request at a time via `rx.blocking_recv()`,
// running each request's whole generate loop inline (the 0.85× regression
// root cause, §0.2). F1 replaces that, for SlotAware only, with a loop
// that ADMITs up to `max_slots` concurrent requests and STEPs the
// scheduler each tick, decoding every active slot per tick.
//
// STEP 1 (this iter) keeps the forward per-slot: `decode tick` loops the
// handles and calls the EXISTING per-slot `forward_decode_slot_aware` /
// `forward_gpu_*` once per handle (still time-sliced — no speedup, by
// design). STEP 2 (F2) swaps that inner loop for one true `[N, hidden]`
// batched forward. The seam is `decode_tick_*` below: F2 changes only how
// the N handles' logits are produced, not the loop/eviction/reply
// machinery here.
//
// Per-arch decode state + tick logic lives BESIDE each arch's serial
// reference (gemma4 here in engine.rs; qwen35 in engine_qwen35.rs) so each
// stays auditable against its `generate_*_once_slot_aware` reference. They
// are deliberately NOT unified: the two arches compute reasoning-token
// counts differently (gemma4 inline during decode; qwen35 post-hoc) and
// handle first-token EOS differently (qwen35 pops + clears; gemma4 does
// not) — unifying would silently change one arch's output.
// ===========================================================================

/// Where a slot's output goes: a unary oneshot (`Request::Generate`) or a
/// streaming SSE channel (`Request::GenerateStream`).
enum SlotReply {
    /// `Request::Generate` — accumulate the full result, fire once at end.
    Unary(oneshot::Sender<Result<GenerationResult>>),
    /// `Request::GenerateStream` — emit a `GenerationEvent::Delta` per
    /// token, a terminal `Done`/`Error` at end. `cancel` is bumped on
    /// client disconnect (mirrors the per-arch streaming serial refs).
    Stream {
        events: mpsc::Sender<super::sse::GenerationEvent>,
        cancel: Option<Arc<std::sync::atomic::AtomicU64>>,
    },
}

/// What one per-arch `decode_tick_*` produced for one slot this tick.
///
/// The arch tick advances exactly one decode token (running the existing
/// per-slot forward + the arch's sample/grammar/stop/EOS tail) and reports
/// the freshly-decoded fragment plus whether the slot is now finished. The
/// shared loop owns reply routing (unary accumulate vs stream `Delta`
/// emit) and, on finish, calls the arch's `finish()` to assemble the
/// arch-divergent `GenerationResult`. Keeping assembly in the arch
/// preserves per-arch reasoning-token counting + `cached_tokens` exactly.
struct TickOutcome {
    /// Freshly-decoded text fragment for this token (empty when the
    /// terminating token is suppressed, e.g. EOS or a stripped stop
    /// string). Streamed as a `Delta` for stream slots.
    fragment: String,
    /// True when the fragment falls inside a reasoning span (streaming
    /// `DeltaKind::Reasoning`). Always false for arches/requests without
    /// reasoning markers.
    is_reasoning: bool,
    /// True once this slot has stopped generating (EOS / max_tokens /
    /// stop-string / grammar-dead). The loop then assembles + fires the
    /// reply and evicts the slot.
    finished: bool,
}

// ---------------------------------------------------------------------------
// gemma4 per-slot decode state + tick (F1 seam; mirrors
// `generate_gemma4_once_slot_aware` engine.rs:8835 exactly).
// ---------------------------------------------------------------------------

/// Hoisted per-slot decode state for a Gemma 4 SlotAware request — the
/// locals that live in the frame of `generate_gemma4_once_slot_aware`'s
/// decode loop, lifted so N slots can interleave across ticks. Field
/// semantics + ordering mirror that function's loop body verbatim so a
/// single slot (N=1) is byte-identical to the serial reference.
struct Gemma4DecodeState {
    slot_id: SlotId,
    prompt_len: usize,
    max_decode_tokens: usize,
    /// `Some` ⇒ slow sampling path (any non-greedy field set); `None` ⇒
    /// greedy fast-path (reuse the forward's on-GPU argmax).
    sampler_params: Option<sampler_pure::SamplingParams>,
    grammar_runtime: Option<super::grammar::GrammarRuntime>,
    /// Shared handle to `params.token_bytes` (the serial ref borrows via
    /// `as_deref()`; the hoist holds the cheap `Arc` clone for the slot's
    /// lifetime). `None` ⇒ no grammar byte-masking.
    token_bytes: Option<Arc<Vec<Vec<u8>>>>,
    tc_splitter: Option<super::registry::ToolCallSplitter>,
    reasoning_splitter: Option<super::registry::ReasoningSplitter>,
    reasoning_enabled: bool,
    reasoning_token_count: usize,
    /// ADR-005 iter-230 B: the request's forced-open seed, carried so
    /// the finish-path full-output re-split starts in the same state
    /// the streaming splitter did.
    reasoning_forced_open: bool,
    want_logprobs: bool,
    logprobs_acc: Option<Vec<f32>>,
    logit_bias: std::collections::HashMap<u32, f32>,
    stop_strings: Vec<String>,
    /// The token to feed into the NEXT decode forward.
    next_token: u32,
    generated_tokens: Vec<u32>,
    decoded_text: String,
    finish_reason: &'static str,
    prefill_duration: Duration,
    decode_started: Instant,
}

impl Gemma4DecodeState {
    /// Run prefill for one Gemma 4 slot and seed the decode state — mirror
    /// of `generate_gemma4_once_slot_aware` engine.rs:8961-9156 (prefill +
    /// sampler/grammar/logprob config + first-token derivation + first
    /// fragment + early-EOS/stop check). On entry the caller has already
    /// done the per-slot KV reset (loop owns reset discipline). Returns the
    /// seeded state; if the first (prefill-emitted) token already
    /// terminates the request, `finish_reason != "length"` and
    /// `generated_tokens` is set accordingly so the loop fires the reply
    /// immediately without a decode tick.
    #[allow(clippy::too_many_arguments)]
    fn prefill_seed(
        loaded: &mut GemmaLoadedModel,
        prompt_tokens: &[u32],
        // Soft-token embedding overrides (multimodal vision path). The
        // `Request::Generate` path passes `&[]` (identity over the
        // text-only prefill); the `Request::GenerateWithSoftTokens` path
        // passes the request's injections — this is the ONLY difference
        // between the two seeds (the underlying primitive already takes
        // soft_tokens), so they share this one fn.
        soft_tokens: &[crate::serve::forward_prefill::SoftTokenInjection<'_>],
        params: &SamplingParams,
        registration: Option<&super::registry::ModelRegistration>,
        slot_id: SlotId,
        multi_seq_kv: &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqHbKvBuffers>,
        mut multi_seq_kv_hybrid: Option<
            &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqHybridKvBuffers>,
        >,
        mut multi_seq_kv_dense: Option<
            &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqDenseKvBuffers>,
        >,
        mut multi_seq_kv_mlx: Option<
            &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqMlxKvCache>,
        >,
    ) -> Result<Self> {
        let max_decode_tokens = params.max_tokens.max(1);

        // Per-slot reset at ENTRY — mirror of the serial ref
        // `generate_gemma4_once_slot_aware` (engine.rs:10386-10412). The
        // persistent multi-seq KV may carry stale bytes from a prior
        // request on this slot OR from spawn-time provisioning (the FIRST
        // request to a slot has had no prior exit-reset). Resetting here is
        // load-bearing for N=1 byte-equivalence: without it the first
        // prefill reads provisioned garbage and diverges from SerialFifo.
        // Resets HB + hybrid (production-default regime); dense/mlx are
        // off-default and their forward paths defense-in-depth on absence.
        for (layer_idx, buf) in multi_seq_kv.iter_mut().enumerate() {
            buf.reset_for_slot(slot_id).map_err(|e| {
                anyhow::anyhow!(
                    "Gemma4DecodeState::prefill_seed: reset_for_slot at entry L{layer_idx}: {e}"
                )
            })?;
        }
        if let Some(hybrid) = multi_seq_kv_hybrid.as_deref_mut() {
            for (layer_idx, buf) in hybrid.iter_mut().enumerate() {
                buf.reset_for_slot(slot_id).map_err(|e| {
                    anyhow::anyhow!(
                        "Gemma4DecodeState::prefill_seed: reset_for_slot at entry (hybrid) \
                         L{layer_idx}: {e}"
                    )
                })?;
            }
        }

        let prefill_started = Instant::now();
        let first_decode_token = loaded.weights.forward_prefill_with_soft_tokens_slot_aware(
            prompt_tokens,
            soft_tokens,
            max_decode_tokens,
            &mut loaded.ctx,
            slot_id,
            multi_seq_kv,
            multi_seq_kv_hybrid.as_deref_mut(),
            multi_seq_kv_dense.as_deref_mut(),
            multi_seq_kv_mlx.as_deref_mut(),
        )?;
        let prefill_duration = prefill_started.elapsed();
        if std::env::var("HF2Q_PREFILL_TIMING").is_ok() {
            eprintln!(
                "[PREFILL_TIMING] slot {} — {} prompt tokens in {:.1} ms ({:.1} prompt tok/s) first_token={}",
                slot_id.0, prompt_tokens.len(),
                prefill_duration.as_secs_f64() * 1000.0,
                prompt_tokens.len() as f64 / prefill_duration.as_secs_f64(),
                first_decode_token,
            );
        }

        // State construction extracted to `from_first_token` (shared with the
        // iter-G(a) batched admit path, which supplies the multi-seq first token).
        Self::from_first_token(
            loaded,
            prompt_tokens,
            params,
            registration,
            slot_id,
            first_decode_token,
            prefill_duration,
        )
    }

    /// ADR-040 iter-G(a) — build the Gemma4DecodeState from an already-computed
    /// prefill first token. Extracted from `prefill_seed` so the cross-slot
    /// batched-admit path (one multi-seq forward → N first tokens) reuses the
    /// identical sampler/grammar/tool-call/reasoning/EOS construction. GREEDY
    /// only is batched (sample_logits==false), so `logits_view()` (read here only
    /// under `if sample_logits`) is never touched for batched callers.
    #[allow(clippy::too_many_arguments)]
    fn from_first_token(
        loaded: &mut GemmaLoadedModel,
        prompt_tokens: &[u32],
        params: &SamplingParams,
        registration: Option<&super::registry::ModelRegistration>,
        slot_id: SlotId,
        first_decode_token: u32,
        prefill_duration: std::time::Duration,
    ) -> Result<Self> {
        let max_decode_tokens = params.max_tokens.max(1);
        // Sampler / grammar / logprobs config — mirror of serial ref
        // engine.rs:9014-9067.
        let sample_logits = params.temperature > 0.0
            || params.top_k > 0
            || params.top_p < 1.0
            || params.repetition_penalty != 1.0
            || !params.logit_bias.is_empty()
            || params.grammar.is_some()
            || params.logprobs;
        let sampler_params = if sample_logits {
            Some(sampler_pure::SamplingParams {
                temperature: params.temperature as f64,
                top_p: params.top_p as f64,
                top_k: params.top_k,
                min_p: 0.0,
                repetition_penalty: effective_repetition_penalty(params),
                max_tokens: params.max_tokens,
            })
        } else {
            None
        };
        let mut grammar_runtime: Option<super::grammar::GrammarRuntime> =
            match params.grammar.as_ref() {
                Some(g) => {
                    let start_rule_id = g
                        .rule_id("root")
                        .ok_or_else(|| anyhow::anyhow!("grammar has no root rule"))?;
                    let mut rt = super::grammar::GrammarRuntime::new(g.clone(), start_rule_id)
                        .ok_or_else(|| anyhow::anyhow!("grammar runtime init failed"))?;
                    if matches!(params.grammar_kind, GrammarKind::ToolCallBodyAuto) {
                        rt.set_awaiting_trigger(true);
                    }
                    Some(rt)
                }
                None => None,
            };
        let token_bytes: Option<Arc<Vec<Vec<u8>>>> = params.token_bytes.clone();
        let mut tc_splitter: Option<super::registry::ToolCallSplitter> =
            registration.and_then(super::registry::ToolCallSplitter::from_registration);
        let want_logprobs = params.logprobs;
        let mut logprobs_acc: Option<Vec<f32>> = if want_logprobs {
            Some(Vec::with_capacity(params.max_tokens))
        } else {
            None
        };

        // First decode token — mirror of serial ref engine.rs:9074-9108.
        let token_bytes_ref: Option<&[Vec<u8>]> = token_bytes.as_deref().map(|v| &v[..]);
        let mut next_token = if sample_logits {
            let sp = sampler_params.as_ref().expect("sample_logits gate");
            let mut logits: Vec<f32> = loaded.weights.logits_view()?.to_vec();
            if !params.logit_bias.is_empty() {
                let v = logits.len();
                for (&id, &bias) in &params.logit_bias {
                    let idx = id as usize;
                    if idx < v {
                        logits[idx] += bias;
                    }
                }
            }
            if let (Some(rt), Some(tb)) = (grammar_runtime.as_ref(), token_bytes_ref) {
                super::grammar::mask::mask_invalid_tokens(rt, tb, &mut logits);
            }
            let (tok, lp_opt) = if want_logprobs {
                let (t, lp) = sampler_pure::sample_token_with_logprob(&mut logits, sp, &[]);
                (t, Some(lp))
            } else {
                (sampler_pure::sample_token(&mut logits, sp, &[]), None)
            };
            if let (Some(acc), Some(lp_val)) = (logprobs_acc.as_mut(), lp_opt) {
                acc.push(lp_val);
            }
            if let (Some(rt), Some(tb)) = (grammar_runtime.as_mut(), token_bytes_ref) {
                let bytes = tb.get(tok as usize).map(|v| v.as_slice()).unwrap_or(&[]);
                if !bytes.is_empty() {
                    rt.accept_bytes(bytes);
                }
            }
            tok
        } else {
            first_decode_token
        };

        let mut reasoning_splitter = registration.filter(|r| r.has_reasoning()).and_then(|r| {
            super::registry::make_reasoning_splitter(r, params.reasoning_forced_open)
        });
        let reasoning_enabled = reasoning_splitter.is_some();
        let reasoning_forced_open = params.reasoning_forced_open;
        let mut reasoning_token_count: usize = 0;

        let decode_started = Instant::now();
        let mut generated_tokens: Vec<u32> = Vec::with_capacity(max_decode_tokens);
        let mut decoded_text = String::new();

        // First emitted token → text BEFORE EOS check (serial ref 9123-9145).
        let first_fragment = loaded
            .tokenizer
            .decode(&[next_token], false)
            .unwrap_or_default();
        decoded_text.push_str(&first_fragment);
        if let Some(sp) = reasoning_splitter.as_mut() {
            let _ = sp.feed(&first_fragment);
            if sp.in_reasoning() {
                reasoning_token_count += 1;
            }
        }
        if let Some(tcs) = tc_splitter.as_mut() {
            let events = tcs.feed(&first_fragment);
            if let Some(rt) = grammar_runtime.as_mut() {
                if events
                    .iter()
                    .any(|e| matches!(e, super::registry::ToolCallEvent::ToolCallOpen))
                {
                    rt.trigger();
                }
            }
        }

        let mut finish_reason: &'static str = "length";
        // Early EOS / stop on the prefill-emitted first token (serial ref
        // 9151-9157). NOTE gemma4 does NOT pop the first token on EOS (it
        // simply does not push it); divergent from qwen35 which pops+clears.
        if loaded.eos_token_ids.contains(&next_token) {
            finish_reason = "stop";
        } else if hit_stop_string(&decoded_text, &params.stop_strings) {
            finish_reason = "stop";
            strip_trailing_stop(&mut decoded_text, &params.stop_strings);
        } else {
            generated_tokens.push(next_token);
        }
        // `next_token` stays as the first token; the first decode tick feeds
        // it (pos = prompt_len + generated_tokens.len() - 1, serial ref 9159).
        let _ = &mut next_token;

        Ok(Gemma4DecodeState {
            slot_id,
            prompt_len: prompt_tokens.len(),
            max_decode_tokens,
            sampler_params,
            grammar_runtime,
            token_bytes,
            tc_splitter,
            reasoning_splitter,
            reasoning_enabled,
            reasoning_token_count,
            reasoning_forced_open,
            want_logprobs,
            logprobs_acc,
            logit_bias: params.logit_bias.clone(),
            stop_strings: params.stop_strings.clone(),
            next_token,
            generated_tokens,
            decoded_text,
            finish_reason,
            prefill_duration,
            decode_started,
        })
    }

    /// Whether the slot already terminated during prefill-seed (first token
    /// was EOS/stop), so the loop should skip decode ticks and finish now.
    fn finished_at_seed(&self) -> bool {
        self.finish_reason != "length"
    }

    /// Advance this slot by exactly one decode token — mirror of the serial
    /// ref's per-token loop body engine.rs:9158-9270 for ONE iteration.
    /// STEP 1: calls the existing per-slot `forward_decode_slot_aware`.
    /// STEP 2 (F2) replaces the caller's per-handle loop with one batched
    /// forward; this body's sample/grammar/stop/EOS tail is unchanged.
    #[allow(clippy::too_many_arguments)]
    fn decode_tick(
        &mut self,
        loaded: &mut GemmaLoadedModel,
        multi_seq_kv: &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqHbKvBuffers>,
        mut multi_seq_kv_hybrid: Option<
            &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqHybridKvBuffers>,
        >,
        mut multi_seq_kv_dense: Option<
            &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqDenseKvBuffers>,
        >,
        mut multi_seq_kv_mlx: Option<
            &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqMlxKvCache>,
        >,
    ) -> Result<TickOutcome> {
        // KV write cursor for the token being fed (serial ref 9159).
        let pos = self.prompt_len + self.generated_tokens.len() - 1;
        let mut p: Option<crate::inference::models::gemma4::profile::TokenProfile> = None;
        let greedy_token = loaded.weights.forward_decode_slot_aware(
            self.next_token,
            pos,
            &mut loaded.ctx,
            &mut p,
            self.slot_id,
            multi_seq_kv,
            multi_seq_kv_hybrid.as_deref_mut(),
            multi_seq_kv_dense.as_deref_mut(),
            multi_seq_kv_mlx.as_deref_mut(),
        )?;

        // ADR-040 S1c-2: the post-forward half (sample/grammar/stop/accumulate)
        // is `decode_tick_finalize`, shared with the batched-head path. Read the
        // slot's logits only when sampling (greedy reuses `greedy_token`).
        let logits: Vec<f32> = if self.sampler_params.is_some() {
            loaded.weights.logits_view()?.to_vec()
        } else {
            Vec::new()
        };
        self.decode_tick_finalize(loaded, greedy_token, &logits)
    }

    /// ADR-040 S1c-2 — post-forward half of [`Self::decode_tick`]: token
    /// selection (greedy → `greedy_token`; sampler → `logits_row` + grammar mask)
    /// then EOS / stop-string / grammar-dead / max_tokens bookkeeping. Shared by
    /// the full per-slot path (logits from `logits_view`) and the batched-head
    /// path in `decode_batch_gemma4` (logits from `lm_head_batched`, greedy from
    /// `finalize_token_from_logits`). Bit-identical token selection either way.
    fn decode_tick_finalize(
        &mut self,
        loaded: &mut GemmaLoadedModel,
        greedy_token: u32,
        logits_row: &[f32],
    ) -> Result<TickOutcome> {
        let token_bytes_ref: Option<&[Vec<u8>]> = self.token_bytes.as_deref().map(|v| &v[..]);
        self.next_token = if let Some(sp) = self.sampler_params.as_ref() {
            let mut logits: Vec<f32> = logits_row.to_vec();
            if !self.logit_bias.is_empty() {
                let v = logits.len();
                for (&id, &bias) in &self.logit_bias {
                    let idx = id as usize;
                    if idx < v {
                        logits[idx] += bias;
                    }
                }
            }
            if let (Some(rt), Some(tb)) = (self.grammar_runtime.as_ref(), token_bytes_ref) {
                super::grammar::mask::mask_invalid_tokens(rt, tb, &mut logits);
            }
            let (tok, lp_opt) = if self.want_logprobs {
                let (t, lp) = sampler_pure::sample_token_with_logprob(
                    &mut logits,
                    sp,
                    &self.generated_tokens,
                );
                (t, Some(lp))
            } else {
                (
                    sampler_pure::sample_token(&mut logits, sp, &self.generated_tokens),
                    None,
                )
            };
            if let (Some(acc), Some(lp_val)) = (self.logprobs_acc.as_mut(), lp_opt) {
                acc.push(lp_val);
            }
            if let (Some(rt), Some(tb)) = (self.grammar_runtime.as_mut(), token_bytes_ref) {
                let bytes = tb.get(tok as usize).map(|v| v.as_slice()).unwrap_or(&[]);
                if !bytes.is_empty() {
                    rt.accept_bytes(bytes);
                }
            }
            tok
        } else {
            greedy_token
        };

        // EOS before push/decode (serial ref 9228-9231): suppress the token.
        if loaded.eos_token_ids.contains(&self.next_token) {
            self.finish_reason = "stop";
            return Ok(TickOutcome {
                fragment: String::new(),
                is_reasoning: false,
                finished: true,
            });
        }
        self.generated_tokens.push(self.next_token);
        let fragment = loaded
            .tokenizer
            .decode(&[self.next_token], false)
            .unwrap_or_default();
        self.decoded_text.push_str(&fragment);
        let mut is_reasoning = false;
        if let Some(sp) = self.reasoning_splitter.as_mut() {
            let _ = sp.feed(&fragment);
            if sp.in_reasoning() {
                self.reasoning_token_count += 1;
                is_reasoning = true;
            }
        }
        if let Some(tcs) = self.tc_splitter.as_mut() {
            let events = tcs.feed(&fragment);
            if let Some(rt) = self.grammar_runtime.as_mut() {
                if events
                    .iter()
                    .any(|e| matches!(e, super::registry::ToolCallEvent::ToolCallOpen))
                {
                    rt.trigger();
                }
            }
        }
        // stop-string (serial ref 9254-9258): the matched suffix is stripped
        // from decoded_text; the just-emitted fragment is still surfaced (the
        // serial ref pushes it before the stop check, identical here).
        if hit_stop_string(&self.decoded_text, &self.stop_strings) {
            self.finish_reason = "stop";
            strip_trailing_stop(&mut self.decoded_text, &self.stop_strings);
            return Ok(TickOutcome {
                fragment,
                is_reasoning,
                finished: true,
            });
        }
        // grammar-dead (serial ref 9262-9270): pop the offending token + re-
        // decode the surviving prefix. The popped token was already emitted
        // as a fragment to a stream client (matches serial ref: the serial
        // ref also fed it to splitters before this check; streaming serial
        // ref behavior is the reference).
        if self.grammar_runtime.as_ref().is_some_and(|rt| rt.is_dead()) {
            self.finish_reason = "stop";
            self.generated_tokens.pop();
            self.decoded_text = loaded
                .tokenizer
                .decode(&self.generated_tokens, false)
                .unwrap_or_default();
            return Ok(TickOutcome {
                fragment,
                is_reasoning,
                finished: true,
            });
        }
        // max_tokens: the scheduler auto-releases at max_tokens, but the
        // generate loop bound is `1..max_decode_tokens` (serial ref 9158).
        // generated_tokens started at 1 (the prefill token); we have emitted
        // `generated_tokens.len()` tokens total. Finish when we reach the
        // bound so finish_reason stays "length".
        let finished = self.generated_tokens.len() >= self.max_decode_tokens;
        Ok(TickOutcome {
            fragment,
            is_reasoning,
            finished,
        })
    }

    /// ADR-040 S1c-2 — body-capture half of [`Self::decode_tick`]: runs the
    /// slot-aware BODY-ONLY decode (same slot-KV isolation as the full path) and
    /// returns this slot's final hidden row `[hidden_size]` (from
    /// `self.activations.hidden`). `decode_batch_gemma4` gathers N slots' rows,
    /// runs ONE `lm_head_batched`, then completes each tick via
    /// [`Self::decode_tick_finalize`]. The hidden must be read here, before the
    /// next slot's capture overwrites the shared `self.activations.hidden`.
    fn decode_tick_capture(
        &mut self,
        loaded: &mut GemmaLoadedModel,
        multi_seq_kv: &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqHbKvBuffers>,
        mut multi_seq_kv_hybrid: Option<
            &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqHybridKvBuffers>,
        >,
        mut multi_seq_kv_dense: Option<
            &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqDenseKvBuffers>,
        >,
        mut multi_seq_kv_mlx: Option<
            &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqMlxKvCache>,
        >,
    ) -> Result<Vec<f32>> {
        let pos = self.prompt_len + self.generated_tokens.len() - 1;
        let mut p: Option<crate::inference::models::gemma4::profile::TokenProfile> = None;
        loaded.weights.forward_decode_slot_aware_capture_hidden(
            self.next_token,
            pos,
            &mut loaded.ctx,
            &mut p,
            self.slot_id,
            multi_seq_kv,
            multi_seq_kv_hybrid.as_deref_mut(),
            multi_seq_kv_dense.as_deref_mut(),
            multi_seq_kv_mlx.as_deref_mut(),
        )?;
        let hs = loaded.weights.hidden_size;
        let hidden: Vec<f32> = loaded
            .weights
            .activations
            .hidden
            .as_slice::<f32>()
            .map_err(|e| anyhow::anyhow!("decode_tick_capture read hidden: {e}"))?
            .get(..hs)
            .ok_or_else(|| anyhow::anyhow!("decode_tick_capture: hidden buffer < hidden_size"))?
            .to_vec();
        Ok(hidden)
    }

    /// Assemble the `GenerationResult` at end-of-decode — mirror of serial
    /// ref engine.rs:9277-9299.
    fn finish(self, registration: Option<&super::registry::ModelRegistration>) -> GenerationResult {
        let (content, reasoning_text) = match registration {
            Some(reg) if reg.has_reasoning() => super::registry::split_full_output_forced(
                reg,
                &self.decoded_text,
                self.reasoning_forced_open,
            ),
            _ => (self.decoded_text, None),
        };
        let decode_duration = self.decode_started.elapsed();
        GenerationResult {
            text: content,
            reasoning_text,
            prompt_tokens: self.prompt_len,
            completion_tokens: self.generated_tokens.len(),
            reasoning_tokens: if self.reasoning_enabled && self.reasoning_token_count > 0 {
                Some(self.reasoning_token_count)
            } else {
                None
            },
            finish_reason: self.finish_reason,
            prefill_duration: self.prefill_duration,
            decode_duration,
            cached_tokens: 0,
            logprobs: self.logprobs_acc,
        }
    }
}

// ---------------------------------------------------------------------------
// The SlotAware worker loop (F1).
// ---------------------------------------------------------------------------

/// ADR-040 Phase F M1 (F1) — the scheduler-driven, admit-while-decoding
/// worker loop for `EngineMode::SlotAware`. Dispatches on the loaded arch
/// (one model per worker) into a per-arch loop that takes the persistent
/// multi-seq KV out of the model ONCE, runs the admit/step/decode/evict
/// cycle, and restores the KV on every exit path via a scope guard.
#[allow(clippy::too_many_arguments)]
fn worker_run_slot_aware(
    loaded: LoadedModel,
    rx: mpsc::Receiver<Request>,
    registration: Option<super::registry::ModelRegistration>,
    max_slots: u32,
    queue_capacity: u32,
    scheduler_stats_snapshot: Arc<Mutex<SchedulerStats>>,
    per_slot_kv_budget_bytes: u64,
    kv_bytes_per_token: u64,
) {
    let scheduler = InflightBatchedScheduler::new_with_kv_budget(
        queue_capacity,
        max_slots,
        per_slot_kv_budget_bytes,
    );
    match loaded {
        LoadedModel::Gemma(g) => run_slot_aware_gemma4(
            g,
            rx,
            registration,
            scheduler,
            scheduler_stats_snapshot,
            per_slot_kv_budget_bytes,
            kv_bytes_per_token,
        ),
        LoadedModel::Qwen35(q) => run_slot_aware_qwen35(
            q,
            rx,
            registration,
            scheduler,
            scheduler_stats_snapshot,
            per_slot_kv_budget_bytes,
            kv_bytes_per_token,
        ),
        // Qwen3-VL text-LM has no slot-aware decode path in M1 scope
        // (ADR-040 §0.5 targets are gemma4 + qwen35moe). Preserve the
        // existing SlotId(N>0) → HTTP 501 behavior: every Generate /
        // GenerateStream is answered with the same typed
        // `capability_unsupported` surface the SerialFifo arm emits at
        // SlotId(N>0), so the operator-facing contract is unchanged.
        LoadedModel::Qwen3VlText(_) => {
            run_slot_aware_qwen3vl_unsupported(rx);
        }
        LoadedModel::Deepseek4(_) => {
            tracing::error!(
                "DeepSeek-V4 reached SlotAware worker despite spawn-time rejection; \
                 rejecting requests through the unsupported worker"
            );
            run_slot_aware_qwen3vl_unsupported(rx);
        }
    }
}

/// Per-token reply routing shared by both arches: a unary slot accumulates;
/// a stream slot emits a `Delta`. Returns `true` if a stream client has
/// disconnected (the loop then evicts the slot + bumps the cancel counter).
fn slot_emit_token(reply: &SlotReply, tick: &TickOutcome) -> bool {
    match reply {
        SlotReply::Unary(_) => false,
        SlotReply::Stream { events, cancel } => {
            if tick.fragment.is_empty() {
                return false;
            }
            let kind = if tick.is_reasoning {
                super::sse::DeltaKind::Reasoning
            } else {
                super::sse::DeltaKind::Content
            };
            if events
                .blocking_send(super::sse::GenerationEvent::Delta {
                    kind,
                    text: tick.fragment.clone(),
                })
                .is_err()
            {
                tracing::info!("SSE stream dropped by client; evicting slot mid-decode");
                if let Some(c) = cancel {
                    c.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
                }
                return true;
            }
            false
        }
    }
}

/// Fire a slot's terminal reply: unary sends the assembled
/// `GenerationResult` over its oneshot; stream sends a `Done` event (unless
/// the client already dropped). `gr` carries the final counts/finish_reason.
fn slot_fire_done(reply: SlotReply, gr: Result<GenerationResult>, client_dropped: bool) {
    match reply {
        SlotReply::Unary(tx) => {
            let _ = tx.send(gr);
        }
        SlotReply::Stream { events, .. } => {
            if client_dropped {
                return;
            }
            match gr {
                Ok(r) => {
                    let _ = events.blocking_send(super::sse::GenerationEvent::Done {
                        finish_reason: r.finish_reason,
                        prompt_tokens: r.prompt_tokens,
                        completion_tokens: r.completion_tokens,
                        stats: super::sse::StreamStats::default(),
                    });
                }
                Err(e) => {
                    let _ =
                        events.blocking_send(super::sse::GenerationEvent::Error(format!("{e:#}")));
                }
            }
        }
    }
}

/// Drain loop for the Qwen3-VL text arch under SlotAware: it has no
/// slot-aware decode path in M1, so every generate request gets the same
/// typed 501 surface the SerialFifo path emits at SlotId(N>0). No KV is
/// taken out (the Qwen3-VL multi-seq scaffold is untouched).
fn run_slot_aware_qwen3vl_unsupported(mut rx: mpsc::Receiver<Request>) {
    while let Some(req) = rx.blocking_recv() {
        match req {
            Request::Generate { reply, .. } => {
                let _ = reply.send(Err(anyhow::anyhow!(
                    "capability_unsupported: ADR-040 Phase F M1 — Qwen3-VL text-LM \
                     has no SlotAware batched-decode path (M1 targets are gemma4 + \
                     qwen35moe per ADR-040 §0.5). Use --scheduler serial-fifo for \
                     this model."
                )));
            }
            Request::GenerateStream { events, .. } => {
                let _ = events.blocking_send(super::sse::GenerationEvent::Error(
                    "capability_unsupported: ADR-040 Phase F M1 — Qwen3-VL text-LM \
                     has no SlotAware batched-decode path (M1 targets are gemma4 + \
                     qwen35moe per ADR-040 §0.5). Use --scheduler serial-fifo for \
                     this model."
                        .to_string(),
                ));
            }
            Request::Warmup { reply } => {
                let _ = reply.send(Ok(()));
            }
            Request::Embed { reply, .. } => {
                let _ = reply.send(Err(anyhow::anyhow!(
                    "capability_unsupported: ADR-040 Phase F M1 — Qwen3-VL Embed \
                     under SlotAware not wired; use serial-fifo."
                )));
            }
            Request::GenerateWithSoftTokens { reply, .. } => {
                let _ = reply.send(Err(anyhow::anyhow!(
                    "capability_unsupported: ADR-040 Phase F M1 — Qwen3-VL \
                     GenerateWithSoftTokens under SlotAware not wired; use serial-fifo."
                )));
            }
            Request::Shutdown => break,
            // Snapshot/restore worker requests are SerialFifo-only control
            // messages; under SlotAware they are not issued. Ignore.
            _ => {}
        }
    }
}

/// Scope guard that holds the Gemma 4 persistent multi-seq KV taken out of
/// the model for the lifetime of the SlotAware loop and restores it on
/// EVERY exit path (normal return, `?`-error, panic unwind) via `Drop`
/// (ADR-040 Phase F M1 lead requirement — NOT per-request take/restore, no
/// `Rc<RefCell>`). The model field stays `None` only while the loop runs;
/// the worker is the sole, serial owner so there is no concurrent access.
struct Gemma4KvGuard<'a> {
    model: &'a mut GemmaLoadedModel,
    kv: Vec<crate::inference::models::gemma4::kv_cache::MultiSeqHbKvBuffers>,
    hybrid: Option<Vec<crate::inference::models::gemma4::kv_cache::MultiSeqHybridKvBuffers>>,
    dense: Option<Vec<crate::inference::models::gemma4::kv_cache::MultiSeqDenseKvBuffers>>,
    mlx: Option<Vec<crate::inference::models::gemma4::kv_cache::MultiSeqMlxKvCache>>,
}

impl<'a> Gemma4KvGuard<'a> {
    /// Take the persistent multi-seq KV (+ Option siblings) out of the
    /// model. Errors if the primary `multi_seq_kv` is absent (the C2c
    /// spawn-arm invariant guarantees it is `Some` for SlotAware Gemma 4).
    fn take(model: &'a mut GemmaLoadedModel) -> Result<Self> {
        let kv = model.multi_seq_kv.take().ok_or_else(|| {
            anyhow::anyhow!(
                "capability_unsupported: ADR-040 Phase F M1 — multi_seq_kv is None \
                 for Gemma 4 SlotAware loop entry. C2c spawn-arm invariant violated \
                 (provision_multi_seq_kv_for_slot_aware not called at SlotAware spawn)."
            )
        })?;
        let hybrid = model.multi_seq_kv_hybrid.take();
        let dense = model.multi_seq_kv_dense.take();
        let mlx = model.multi_seq_kv_mlx.take();
        Ok(Self {
            model,
            kv,
            hybrid,
            dense,
            mlx,
        })
    }
}

impl Drop for Gemma4KvGuard<'_> {
    fn drop(&mut self) {
        // Restore the taken buffers so the model's spawn-time invariant
        // (`multi_seq_kv.is_some()` for SlotAware Gemma 4) holds for any
        // future use of `loaded`. `std::mem::take` leaves empty/None in the
        // guard fields; the model fields are overwritten wholesale.
        self.model.multi_seq_kv = Some(std::mem::take(&mut self.kv));
        self.model.multi_seq_kv_hybrid = self.hybrid.take();
        self.model.multi_seq_kv_dense = self.dense.take();
        self.model.multi_seq_kv_mlx = self.mlx.take();
    }
}

/// Type alias for one installed Gemma 4 slot: its decode state, where its
/// output goes, and its scheduler handle (for `advance_after_decode` /
/// `release`).
type Gemma4Slot = (Gemma4DecodeState, SlotReply, SlotHandle);

/// ADR-040 Phase F M1 (F1) — Gemma 4 SlotAware admit-while-decoding loop.
///
/// One worker thread, up to `max_slots` concurrent requests. Each tick:
/// ADMIT new requests into free slots (running their prefill now), STEP the
/// scheduler, then DECODE every active slot once (STEP 1: per-slot forward
/// in a loop — no batched forward yet). A slot that finishes (EOS /
/// max_tokens / stop / grammar-dead / client-drop) fires its reply and is
/// evicted mid-batch without stalling peers; the freed slot refills on the
/// next admit. N=1 is byte-identical to `generate_gemma4_once_slot_aware`.
#[allow(clippy::too_many_arguments)]
fn run_slot_aware_gemma4(
    mut model: GemmaLoadedModel,
    mut rx: mpsc::Receiver<Request>,
    registration: Option<super::registry::ModelRegistration>,
    mut scheduler: InflightBatchedScheduler,
    scheduler_stats_snapshot: Arc<Mutex<SchedulerStats>>,
    per_slot_kv_budget_bytes: u64,
    kv_bytes_per_token: u64,
) {
    let mut guard = match Gemma4KvGuard::take(&mut model) {
        Ok(g) => g,
        Err(e) => {
            tracing::error!("Gemma4 SlotAware loop cannot start: {e:#}");
            // Drain with a typed error so callers do not hang.
            drain_with_startup_error(rx, "Gemma 4 multi_seq_kv absent at SlotAware entry");
            return;
        }
    };
    let n_slots = guard.kv.first().map(|b| b.n_seqs).unwrap_or(0) as usize;
    let mut slots: Vec<Option<Gemma4Slot>> = (0..n_slots).map(|_| None).collect();

    let publish = |sched: &InflightBatchedScheduler, snap: &Arc<Mutex<SchedulerStats>>| {
        if let Ok(mut g) = snap.lock() {
            *g = sched.stats();
        }
    };

    // A request pulled off `rx` that still needs admitting (set when the
    // loop parked on Idle and blocked for one request).
    let mut pending: Option<Request> = None;

    // ADR-040 iter-G(a) — cross-slot BATCHED admit gate. When on, the admit
    // phase collects greedy text requests for free slots and prefills them in
    // ONE multi-seq forward (the TTFT lever). Opt-in (HF2Q_CROSS_SLOT_ADMIT=1)
    // + capability-gated (hybrid-KV regime, scaffold present, no BF16-xlen
    // verify cache). When off OR unsupported, the admit phase is BYTE-UNCHANGED
    // (the original one-request-per-slot loop). Stable across the worker's life.
    let cross_slot_admit = std::env::var("HF2Q_CROSS_SLOT_ADMIT").as_deref() == Ok("1")
        && guard.hybrid.is_some()
        && crate::debug::INVESTIGATION_ENV.hybrid_kv
        && std::env::var("HF2Q_DFLASH_XLEN_SDPA").as_deref() != Ok("1");

    // ADR-040 production profiling — zero the buckets at worker entry so the
    // worker-exit dump reflects THIS worker's lifetime, not leftover state.
    // Mirrors the throughput-probe test fn's reset-before-timed-run pattern.
    crate::inference::models::gemma4::batched_body::catsplit::reset();
    crate::inference::models::gemma4::batched_body::host_phases::reset();

    'worker: loop {
        let _hp_iter = std::time::Instant::now();
        // ── ADMIT ────────────────────────────────────────────────────
        // Fill free slots without blocking. The first iteration may carry
        // a `pending` request that unparked us.
        if cross_slot_admit {
            // ADR-040 iter-G(a) — BATCHED admit. Drain requests for the free
            // slots, peel greedy text into ONE multi-seq prefill; dispatch
            // everything else (sampling / soft-tokens / warmup / embed) via the
            // single path, exactly as the default loop does.
            loop {
                let n_free = slots.iter().filter(|s| s.is_none()).count();
                if n_free == 0 {
                    break;
                }
                let mut reqs: Vec<Request> = Vec::with_capacity(n_free);
                while reqs.len() < n_free {
                    let req = match pending.take() {
                        Some(r) => r,
                        None => match rx.try_recv() {
                            Ok(r) => r,
                            Err(mpsc::error::TryRecvError::Empty) => break,
                            Err(mpsc::error::TryRecvError::Disconnected) => break 'worker,
                        },
                    };
                    reqs.push(req);
                }
                if reqs.is_empty() {
                    break;
                }
                let mut batch: Vec<(Vec<u32>, SamplingParams, SlotReply)> = Vec::new();
                for req in reqs {
                    match req {
                        Request::Generate {
                            prompt_tokens,
                            params,
                            reply,
                        } if params_is_greedy(&params) && params.max_tokens > 0 => {
                            batch.push((prompt_tokens, params, SlotReply::Unary(reply)));
                        }
                        Request::GenerateStream {
                            prompt_tokens,
                            params,
                            events,
                            cancellation_counter,
                            ..
                        } if params_is_greedy(&params) && params.max_tokens > 0 => {
                            batch.push((
                                prompt_tokens,
                                params,
                                SlotReply::Stream {
                                    events,
                                    cancel: cancellation_counter,
                                },
                            ));
                        }
                        Request::Generate {
                            prompt_tokens,
                            params,
                            reply,
                        } => {
                            admit_gemma4_slot(
                                &mut guard,
                                &mut scheduler,
                                &mut slots,
                                registration.as_ref(),
                                &scheduler_stats_snapshot,
                                per_slot_kv_budget_bytes,
                                kv_bytes_per_token,
                                prompt_tokens,
                                Vec::new(),
                                params,
                                SlotReply::Unary(reply),
                            );
                            publish(&scheduler, &scheduler_stats_snapshot);
                        }
                        Request::GenerateStream {
                            prompt_tokens,
                            params,
                            events,
                            cancellation_counter,
                            ..
                        } => {
                            admit_gemma4_slot(
                                &mut guard,
                                &mut scheduler,
                                &mut slots,
                                registration.as_ref(),
                                &scheduler_stats_snapshot,
                                per_slot_kv_budget_bytes,
                                kv_bytes_per_token,
                                prompt_tokens,
                                Vec::new(),
                                params,
                                SlotReply::Stream {
                                    events,
                                    cancel: cancellation_counter,
                                },
                            );
                            publish(&scheduler, &scheduler_stats_snapshot);
                        }
                        Request::GenerateWithSoftTokens {
                            prompt_tokens,
                            soft_tokens,
                            params,
                            reply,
                            ..
                        } => {
                            admit_gemma4_slot(
                                &mut guard,
                                &mut scheduler,
                                &mut slots,
                                registration.as_ref(),
                                &scheduler_stats_snapshot,
                                per_slot_kv_budget_bytes,
                                kv_bytes_per_token,
                                prompt_tokens,
                                soft_tokens,
                                params,
                                SlotReply::Unary(reply),
                            );
                            publish(&scheduler, &scheduler_stats_snapshot);
                        }
                        Request::Warmup { reply } => {
                            let _ = reply.send(warmup_once(&mut guard.model));
                        }
                        Request::Shutdown => {
                            tracing::info!("Gemma4 SlotAware worker received Shutdown; exiting");
                            break 'worker;
                        }
                        Request::Embed {
                            prompt_tokens,
                            reply,
                        } => {
                            embed_gemma4_inline(
                                &mut guard,
                                &mut scheduler,
                                &scheduler_stats_snapshot,
                                per_slot_kv_budget_bytes,
                                kv_bytes_per_token,
                                prompt_tokens,
                                reply,
                            );
                            publish(&scheduler, &scheduler_stats_snapshot);
                        }
                        _ => {}
                    }
                }
                if batch.len() >= 2 {
                    admit_gemma4_slots_batched(
                        &mut guard,
                        &mut scheduler,
                        &mut slots,
                        registration.as_ref(),
                        &scheduler_stats_snapshot,
                        per_slot_kv_budget_bytes,
                        kv_bytes_per_token,
                        batch,
                    );
                    publish(&scheduler, &scheduler_stats_snapshot);
                } else {
                    for (prompt_tokens, params, reply) in batch {
                        admit_gemma4_slot(
                            &mut guard,
                            &mut scheduler,
                            &mut slots,
                            registration.as_ref(),
                            &scheduler_stats_snapshot,
                            per_slot_kv_budget_bytes,
                            kv_bytes_per_token,
                            prompt_tokens,
                            Vec::new(),
                            params,
                            reply,
                        );
                        publish(&scheduler, &scheduler_stats_snapshot);
                    }
                }
            }
        } else {
            loop {
                let free = slots.iter().position(|s| s.is_none());
                let Some(free_idx) = free else { break };
                let _ = free_idx;
                let req = match pending.take() {
                    Some(r) => r,
                    None => match rx.try_recv() {
                        Ok(r) => r,
                        Err(mpsc::error::TryRecvError::Empty) => break,
                        Err(mpsc::error::TryRecvError::Disconnected) => break 'worker,
                    },
                };
                match req {
                    Request::Generate {
                        prompt_tokens,
                        params,
                        reply,
                    } => {
                        admit_gemma4_slot(
                            &mut guard,
                            &mut scheduler,
                            &mut slots,
                            registration.as_ref(),
                            &scheduler_stats_snapshot,
                            per_slot_kv_budget_bytes,
                            kv_bytes_per_token,
                            prompt_tokens,
                            Vec::new(),
                            params,
                            SlotReply::Unary(reply),
                        );
                        publish(&scheduler, &scheduler_stats_snapshot);
                    }
                    Request::GenerateStream {
                        prompt_tokens,
                        params,
                        events,
                        cancellation_counter,
                        ..
                    } => {
                        admit_gemma4_slot(
                            &mut guard,
                            &mut scheduler,
                            &mut slots,
                            registration.as_ref(),
                            &scheduler_stats_snapshot,
                            per_slot_kv_budget_bytes,
                            kv_bytes_per_token,
                            prompt_tokens,
                            Vec::new(),
                            params,
                            SlotReply::Stream {
                                events,
                                cancel: cancellation_counter,
                            },
                        );
                        publish(&scheduler, &scheduler_stats_snapshot);
                    }
                    // GenerateWithSoftTokens (multimodal vision generation) IS
                    // served at SlotId>0 by the legacy inflight arm today
                    // (generate_gemma4_once_with_soft_tokens_slot_aware,
                    // engine.rs:8231), so 501-ing it would be a regression. It
                    // is a generation → it joins the batched decode loop exactly
                    // like Generate, the only difference being the soft-token
                    // injections threaded into the prefill seed. (deepstack /
                    // positions_flat are Qwen3-VL-only; Gemma 4 ignores them,
                    // matching the legacy arm at engine.rs:8219-8229.)
                    Request::GenerateWithSoftTokens {
                        prompt_tokens,
                        soft_tokens,
                        params,
                        reply,
                        ..
                    } => {
                        admit_gemma4_slot(
                            &mut guard,
                            &mut scheduler,
                            &mut slots,
                            registration.as_ref(),
                            &scheduler_stats_snapshot,
                            per_slot_kv_budget_bytes,
                            kv_bytes_per_token,
                            prompt_tokens,
                            soft_tokens,
                            params,
                            SlotReply::Unary(reply),
                        );
                        publish(&scheduler, &scheduler_stats_snapshot);
                    }
                    Request::Warmup { reply } => {
                        let _ = reply.send(warmup_once(&mut guard.model));
                    }
                    Request::Shutdown => {
                        tracing::info!("Gemma4 SlotAware worker received Shutdown; exiting");
                        break 'worker;
                    }
                    // Embed (pooled last-token embedding) IS served at SlotId>0
                    // by the legacy inflight arm today (embed_gemma4_slot_aware,
                    // engine.rs:7839), so 501-ing it would be a regression. It is
                    // a one-shot (no decode loop) → run it inline in the admit
                    // path (like Warmup): reserve a slot, run the slot-aware
                    // embed, release.
                    Request::Embed {
                        prompt_tokens,
                        reply,
                    } => {
                        embed_gemma4_inline(
                            &mut guard,
                            &mut scheduler,
                            &scheduler_stats_snapshot,
                            per_slot_kv_budget_bytes,
                            kv_bytes_per_token,
                            prompt_tokens,
                            reply,
                        );
                        publish(&scheduler, &scheduler_stats_snapshot);
                    }
                    _ => {}
                }
            }
        }

        // ── STEP ─────────────────────────────────────────────────────
        let _hp_sched = std::time::Instant::now();
        let step = match scheduler.step() {
            Ok(s) => s,
            Err(e) => {
                tracing::error!("Gemma4 SlotAware scheduler.step() failed: {e:?}");
                break 'worker;
            }
        };
        crate::inference::models::gemma4::batched_body::host_phases::add(
            crate::inference::models::gemma4::batched_body::host_phases::Phase::SchedStep,
            _hp_sched.elapsed().as_nanos() as u64,
        );
        match step {
            SchedulerStep::Idle => {
                // No work. Park until a request arrives, then re-loop to
                // admit it. Disconnect ends the worker.
                match rx.blocking_recv() {
                    Some(r) => pending = Some(r),
                    None => break 'worker,
                }
            }
            SchedulerStep::Prefill { .. } => {
                // Prefill is run eagerly at admit time (prefill_seed), so the
                // scheduler's Prefilling phase is advanced immediately there.
                // Reaching here means a slot is still Prefilling in the
                // scheduler's view but has no pending forward work — advance
                // is handled in admit. Nothing to do this tick.
            }
            SchedulerStep::Decode { handles } => {
                let _hp_db = std::time::Instant::now();
                decode_batch_gemma4(
                    &mut guard,
                    &mut scheduler,
                    &mut slots,
                    registration.as_ref(),
                    &handles,
                );
                crate::inference::models::gemma4::batched_body::host_phases::add(
                    crate::inference::models::gemma4::batched_body::host_phases::Phase::DecodeBatchTotal,
                    _hp_db.elapsed().as_nanos() as u64,
                );
                let _hp_pub = std::time::Instant::now();
                publish(&scheduler, &scheduler_stats_snapshot);
                crate::inference::models::gemma4::batched_body::host_phases::add(
                    crate::inference::models::gemma4::batched_body::host_phases::Phase::Publish,
                    _hp_pub.elapsed().as_nanos() as u64,
                );
            }
            SchedulerStep::Mixed { decode_handles, .. } => {
                // Prefill already ran at admit; the just-admitted slot is
                // Prefilling and excluded from `decode_handles`
                // (collect_decoding_handles skips Prefilling). Decode the
                // rest this tick.
                decode_batch_gemma4(
                    &mut guard,
                    &mut scheduler,
                    &mut slots,
                    registration.as_ref(),
                    &decode_handles,
                );
                let _hp_pub = std::time::Instant::now();
                publish(&scheduler, &scheduler_stats_snapshot);
                crate::inference::models::gemma4::batched_body::host_phases::add(
                    crate::inference::models::gemma4::batched_body::host_phases::Phase::Publish,
                    _hp_pub.elapsed().as_nanos() as u64,
                );
            }
        }
        crate::inference::models::gemma4::batched_body::host_phases::add(
            crate::inference::models::gemma4::batched_body::host_phases::Phase::WorkerIter,
            _hp_iter.elapsed().as_nanos() as u64,
        );
    }

    // ADR-040 production profiling emit — when HF2Q_DECODE_CATSPLIT=1 or
    // HF2Q_HOST_PHASES=1, dump the accumulated buckets at worker-thread exit.
    // Mirrors the throughput-probe test fn's [CATSPLIT]/[HOST_PHASES] tables
    // but reports WORKER-LIFETIME totals (not per-token/per-step) since the
    // worker accumulates across its entire run.
    if *crate::inference::models::gemma4::batched_body::catsplit::ENABLED {
        let snap = crate::inference::models::gemma4::batched_body::catsplit::snapshot();
        let mut rows: Vec<(&str, u64, u64, u64)> = snap
            .into_iter()
            .filter(|(_, ns, _, disp)| *ns > 0 || *disp > 0)
            .collect();
        let sum_ns: u64 = rows.iter().map(|(_, ns, _, _)| *ns).sum::<u64>().max(1);
        let sum_disp: u64 = rows.iter().map(|(_, _, _, d)| *d).sum::<u64>();
        rows.sort_by(|a, b| b.3.cmp(&a.3));
        eprintln!(
            "[CATSPLIT] worker-exit lifetime dump: {} categories, {} total dispatches:",
            rows.len(),
            sum_disp,
        );
        eprintln!(
            "[CATSPLIT]   {:<40} {:>12} {:>8} {:>12} {:>10}",
            "category", "total_ms", "% step", "total_disp", "total_cbs",
        );
        for (name, ns, cbs, disp) in &rows {
            eprintln!(
                "[CATSPLIT]   {:<40} {:>12.3} {:>7.1}% {:>12} {:>10}",
                name,
                *ns as f64 / 1e6,
                100.0 * *ns as f64 / sum_ns as f64,
                disp,
                cbs,
            );
        }
        eprintln!(
            "[CATSPLIT]   {:<40} {:>12.3} {:>7.1}% (category-sum; CB-serialized, > real overlapped step)",
            "TOTAL",
            sum_ns as f64 / 1e6,
            100.0,
        );
    }
    if *crate::inference::models::gemma4::batched_body::host_phases::ENABLED {
        let hp = crate::inference::models::gemma4::batched_body::host_phases::snapshot();
        if hp.iter().any(|(_, ns)| *ns > 0) {
            let leaf = hp.len().saturating_sub(2);
            let total: u64 = hp.iter().take(leaf).map(|(_, ns)| *ns).sum();
            eprintln!("[HOST_PHASES] worker-exit lifetime dump:");
            for (i, (name, ns)) in hp.iter().enumerate() {
                let tag = if i >= leaf { " [ref]" } else { "" };
                eprintln!(
                    "[HOST_PHASES]   {:<32} {:>10.3} ms ({:5.1}%){}",
                    name,
                    *ns as f64 / 1e6,
                    100.0 * *ns as f64 / total.max(1) as f64,
                    tag,
                );
            }
            eprintln!(
                "[HOST_PHASES]   {:<32} {:>10.3} ms (sum of leaf phases)",
                "TOTAL",
                total as f64 / 1e6,
            );
        }
    }

    // Guard drops here → KV restored into `model` on every exit path.
    drop(guard);
    tracing::info!("Gemma4 SlotAware worker thread exited");
}

/// Admit one Gemma 4 request: reserve a scheduler slot, run prefill now
/// (`Gemma4DecodeState::prefill_seed`), and either fire immediately (first
/// token already terminal) or install the slot for decode ticks.
#[allow(clippy::too_many_arguments)]
fn admit_gemma4_slot(
    guard: &mut Gemma4KvGuard<'_>,
    scheduler: &mut InflightBatchedScheduler,
    slots: &mut [Option<Gemma4Slot>],
    registration: Option<&super::registry::ModelRegistration>,
    scheduler_stats_snapshot: &Arc<Mutex<SchedulerStats>>,
    per_slot_kv_budget_bytes: u64,
    kv_bytes_per_token: u64,
    prompt_tokens: Vec<u32>,
    // Owned soft-token injections (empty for plain Generate/GenerateStream;
    // the request's vision embeddings for GenerateWithSoftTokens). Borrowed
    // into `SoftTokenInjection` slices for the prefill below.
    soft_token_data: Vec<SoftTokenData>,
    params: SamplingParams,
    reply: SlotReply,
) {
    // Build borrowed injection slices from the owned data — same shape as
    // the legacy SoftTokens arm (engine.rs:8212-8218). Empty ⇒ identity
    // over the text-only prefill (Generate path).
    let soft_tokens: Vec<crate::serve::forward_prefill::SoftTokenInjection<'_>> = soft_token_data
        .iter()
        .map(|d| crate::serve::forward_prefill::SoftTokenInjection {
            range: d.range.clone(),
            embeddings: &d.embeddings,
        })
        .collect();
    let needed_bytes: u64 = if kv_bytes_per_token == 0 || per_slot_kv_budget_bytes == 0 {
        0
    } else {
        u64::from(prompt_tokens.len() as u32)
            .saturating_add(u64::from(params.max_tokens as u32))
            .saturating_mul(kv_bytes_per_token)
    };
    let admit_req = AdmitRequest {
        prompt_tokens: prompt_tokens.len() as u32,
        max_tokens: params.max_tokens as u32,
        kv_bytes_needed: needed_bytes,
    };
    let admitted = match scheduler.admit(admit_req) {
        Ok(slot) => slot,
        Err(AdmitError::SlotBudgetExceeded {
            needed_bytes,
            budget_bytes,
        }) => {
            let gr = Err(anyhow::anyhow!(
                "slot_budget_exceeded: ADR-040 Phase F M1 — per-slot KV budget exceeded \
                 (needed_bytes={needed_bytes}, budget_bytes={budget_bytes}). Reduce \
                 max_tokens or use a shorter prompt."
            ));
            slot_fire_done(reply, gr, false);
            return;
        }
        Err(e) => {
            let gr = Err(anyhow::anyhow!("ADR-040 Phase F M1 admit failed: {e:?}"));
            slot_fire_done(reply, gr, false);
            return;
        }
    };
    let Some(handle) = admitted.handle else {
        // `handle: None` == the scheduler's CompletedAtAdmit outcome
        // (`max_tokens == 0`): no physical slot allocated, no scheduler
        // bookkeeping. Mirror the SerialFifo arm's fallback EXACTLY
        // (engine.rs `else` of `if let Some(handle) = admitted.handle`):
        // run the legacy NON-slot-aware generate (which applies
        // `max_tokens.max(1)` → one decode token) and fire. With soft
        // tokens present, the soft-token sibling is used. Preserves
        // byte-equivalence for the degenerate `max_tokens == 0` request.
        let gr = if soft_tokens.is_empty() {
            generate_once(guard.model, &prompt_tokens, &params, registration)
        } else {
            generate_once_with_soft_tokens(
                guard.model,
                &prompt_tokens,
                &soft_tokens,
                &params,
                registration,
            )
        };
        slot_fire_done(reply, gr, false);
        return;
    };

    // Restore the per-prefill self-mount fresh state (None) before this
    // slot's prefill, so a prior slot's leftover slice-view does not poison
    // this slot's consume-gate (forward_prefill.rs:699/2344; precedent at
    // forward_embed_last :2459). Load-bearing for N>1 concurrency.
    clear_gemma4_self_mounts(guard.model);

    let seed = Gemma4DecodeState::prefill_seed(
        guard.model,
        &prompt_tokens,
        &soft_tokens,
        &params,
        registration,
        handle.slot_id,
        &mut guard.kv,
        guard.hybrid.as_mut(),
        guard.dense.as_mut(),
        guard.mlx.as_mut(),
    );
    let state = match seed {
        Ok(s) => s,
        Err(e) => {
            // Prefill failed: reset the slot's KV, release, fire error.
            reset_gemma4_slot(guard, handle.slot_id);
            scheduler.release(handle);
            if let Ok(mut g) = scheduler_stats_snapshot.lock() {
                *g = scheduler.stats();
            }
            slot_fire_done(reply, Err(e), false);
            return;
        }
    };

    // ADR-040 Phase F `iter-F-prefill-determinism` (2026-06-24) — clear the
    // per-prefill self-mounts AGAIN, now AFTER prefill_seed, to enforce the
    // postcondition "the SlotAware worker leaves `self.{dense,hybrid,leg_hb}_kv`
    // == None between requests". The slot-aware prefill mounts a per-slot
    // slice-view on these shared fields and (in the current forward) restores
    // the PRIOR value on exit; the per-slot decode uses a save-mount-RESTORE
    // scope-guard. When a NEW request is admitted mid-stream (the prefilled K/V
    // already lands durably in the per-slot `multi_seq_kv_hybrid` scaffold, so
    // this is data-lossless), any prefill-origin mount that survives on `self.*`
    // gets RESTORED by the next in-flight slot's decode and then poisons the
    // following prefill's `if self.hybrid_kv.is_none()` write-back gate
    // (forward_prefill.rs:970) — corrupting the earliest in-flight request
    // (root-caused via `slot_aware_staggered_eviction`: ~17% gross corruption of
    // the first slot, codex-confirmed). Mirrors the pre-prefill clear above.
    clear_gemma4_self_mounts(guard.model);

    // Prefill consumed the whole prompt in one shot → advance the
    // scheduler's Prefilling phase to Decoding immediately.
    scheduler.advance_after_prefill(handle, prompt_tokens.len() as u32);

    if state.finished_at_seed() {
        // First (prefill-emitted) token already terminated. Emit the decoded
        // first-token text (stream) then fire the terminal reply; no decode
        // tick. For gemma4, on first-token EOS the token is NOT pushed but
        // its text WAS appended (serial ref 9151-9155 only strips on stop-
        // string); emit whatever decoded_text holds.
        let seed_tick = TickOutcome {
            fragment: state.decoded_text.clone(),
            is_reasoning: false,
            finished: true,
        };
        let dropped = slot_emit_token(&reply, &seed_tick);
        scheduler.advance_after_decode(handle);
        reset_gemma4_slot(guard, handle.slot_id);
        scheduler.release(handle);
        let gr = Ok(state.finish(registration));
        slot_fire_done(reply, gr, dropped);
        return;
    }

    let slot_idx = handle.slot_id.0 as usize;
    slots[slot_idx] = Some((state, reply, handle));
}

/// ADR-040 iter-G(a) — is this request GREEDY (`sample_logits == false`)? Only
/// greedy text requests are cross-slot batchable: the multi-seq forward returns
/// per-seq ARGMAX first tokens (not per-seq logits), and `from_first_token`
/// reads `logits_view()` ONLY under `sample_logits`. Mirror of the predicate in
/// `prefill_seed`/`from_first_token`.
fn params_is_greedy(p: &SamplingParams) -> bool {
    !(p.temperature > 0.0
        || p.top_k > 0
        || p.top_p < 1.0
        || p.repetition_penalty != 1.0
        || !p.logit_bias.is_empty()
        || p.grammar.is_some()
        || p.logprobs)
}

/// ADR-040 iter-G(a) — cross-slot BATCHED admit. Prefills N greedy text requests
/// in ONE multi-seq forward (`forward_prefill_batched_multi_seq`) instead of N
/// sequential single-seq prefills — the short-prompt N-concurrent TTFT lever.
/// Caller guarantees every request is greedy text with `max_tokens > 0` and the
/// hybrid-KV multi-seq regime is supported (capability-gated). All-or-nothing on
/// the forward; per-request admit failures fall back to the single path.
///
/// `ITER_GA_BATCHED_ADMIT_COUNT` counts forwards with N>=2 (E2E test signal).
pub(crate) static ITER_GA_BATCHED_ADMIT_COUNT: std::sync::atomic::AtomicUsize =
    std::sync::atomic::AtomicUsize::new(0);

#[allow(clippy::too_many_arguments)]
fn admit_gemma4_slots_batched(
    guard: &mut Gemma4KvGuard<'_>,
    scheduler: &mut InflightBatchedScheduler,
    slots: &mut [Option<Gemma4Slot>],
    registration: Option<&super::registry::ModelRegistration>,
    scheduler_stats_snapshot: &Arc<Mutex<SchedulerStats>>,
    per_slot_kv_budget_bytes: u64,
    kv_bytes_per_token: u64,
    requests: Vec<(Vec<u32>, SamplingParams, SlotReply)>,
) {
    // 1. Reserve a physical slot for each request.
    let mut admitted: Vec<(SlotHandle, Vec<u32>, SamplingParams, SlotReply)> =
        Vec::with_capacity(requests.len());
    for (prompt_tokens, params, reply) in requests {
        let needed_bytes: u64 = if kv_bytes_per_token == 0 || per_slot_kv_budget_bytes == 0 {
            0
        } else {
            u64::from(prompt_tokens.len() as u32)
                .saturating_add(u64::from(params.max_tokens as u32))
                .saturating_mul(kv_bytes_per_token)
        };
        let admit_req = AdmitRequest {
            prompt_tokens: prompt_tokens.len() as u32,
            max_tokens: params.max_tokens as u32,
            kv_bytes_needed: needed_bytes,
        };
        match scheduler.admit(admit_req) {
            Ok(a) => match a.handle {
                Some(h) => admitted.push((h, prompt_tokens, params, reply)),
                // handle:None == queued/completed-at-admit. We only batch when
                // slots are free + max_tokens>0, so this is not expected; serve
                // via the legacy non-slot path rather than drop the request.
                None => {
                    let gr = generate_once(guard.model, &prompt_tokens, &params, registration);
                    slot_fire_done(reply, gr, false);
                }
            },
            Err(e) => slot_fire_done(
                reply,
                Err(anyhow::anyhow!(
                    "ADR-040 iter-G(a) batched admit failed: {e:?}"
                )),
                false,
            ),
        }
    }
    if let Ok(mut g) = scheduler_stats_snapshot.lock() {
        *g = scheduler.stats();
    }

    // Fewer than 2 survived → the batched forward isn't worthwhile; route the
    // survivor(s) through the validated single path (release the reservation
    // first; admit_gemma4_slot re-admits).
    if admitted.len() < 2 {
        for (handle, prompt_tokens, params, reply) in admitted {
            scheduler.release(handle);
            admit_gemma4_slot(
                guard,
                scheduler,
                slots,
                registration,
                scheduler_stats_snapshot,
                per_slot_kv_budget_bytes,
                kv_bytes_per_token,
                prompt_tokens,
                Vec::new(),
                params,
                reply,
            );
        }
        return;
    }

    // 2. Per-slot entry reset (kv + hybrid) + clear self-mounts (mirror
    //    prefill_seed's entry reset + the iter-F-prefill-determinism pre-clear).
    for (handle, _, _, _) in &admitted {
        reset_gemma4_slot(guard, handle.slot_id);
    }
    clear_gemma4_self_mounts(guard.model);

    ITER_GA_BATCHED_ADMIT_COUNT.fetch_add(1, std::sync::atomic::Ordering::Relaxed);

    // 3. ONE multi-seq forward → N first tokens (writes each slot's KV scatter).
    let seqs: Vec<(Vec<u32>, SlotId)> = admitted
        .iter()
        .map(|(h, p, _, _)| (p.clone(), h.slot_id))
        .collect();
    let max_decode = admitted
        .iter()
        .map(|(_, _, params, _)| params.max_tokens.max(1))
        .max()
        .unwrap_or(1);
    let prefill_started = Instant::now();
    let scaffold = guard
        .hybrid
        .as_deref()
        .expect("iter-G(a): hybrid scaffold present (capability-gated by caller)");
    let forward = guard.model.weights.forward_prefill_batched_multi_seq(
        &seqs,
        scaffold,
        max_decode,
        &mut guard.model.ctx,
    );
    let prefill_duration = prefill_started.elapsed();
    clear_gemma4_self_mounts(guard.model);

    let tokens = match forward {
        Ok(t) => t,
        Err(e) => {
            // All-or-nothing: reset + release + error every reserved slot.
            let msg = format!("ADR-040 iter-G(a) multi-seq prefill failed: {e}");
            for (handle, _, _, reply) in admitted {
                reset_gemma4_slot(guard, handle.slot_id);
                scheduler.release(handle);
                slot_fire_done(reply, Err(anyhow::anyhow!("{msg}")), false);
            }
            if let Ok(mut g) = scheduler_stats_snapshot.lock() {
                *g = scheduler.stats();
            }
            return;
        }
    };
    if std::env::var("HF2Q_PREFILL_TIMING").is_ok() {
        eprintln!(
            "[PREFILL_TIMING] BATCHED {} seqs in {:.1} ms (one multi-seq forward, iter-G(a))",
            seqs.len(),
            prefill_duration.as_secs_f64() * 1000.0,
        );
    }

    // 4. Install each slot — identical tail to admit_gemma4_slot.
    for ((handle, prompt_tokens, params, reply), first_token) in
        admitted.into_iter().zip(tokens.into_iter())
    {
        let state = match Gemma4DecodeState::from_first_token(
            guard.model,
            &prompt_tokens,
            &params,
            registration,
            handle.slot_id,
            first_token,
            prefill_duration,
        ) {
            Ok(s) => s,
            Err(e) => {
                reset_gemma4_slot(guard, handle.slot_id);
                scheduler.release(handle);
                slot_fire_done(reply, Err(e), false);
                continue;
            }
        };
        scheduler.advance_after_prefill(handle, prompt_tokens.len() as u32);
        if state.finished_at_seed() {
            let seed_tick = TickOutcome {
                fragment: state.decoded_text.clone(),
                is_reasoning: false,
                finished: true,
            };
            let dropped = slot_emit_token(&reply, &seed_tick);
            scheduler.advance_after_decode(handle);
            reset_gemma4_slot(guard, handle.slot_id);
            scheduler.release(handle);
            let gr = Ok(state.finish(registration));
            slot_fire_done(reply, gr, dropped);
            continue;
        }
        let slot_idx = handle.slot_id.0 as usize;
        slots[slot_idx] = Some((state, reply, handle));
    }
    if let Ok(mut g) = scheduler_stats_snapshot.lock() {
        *g = scheduler.stats();
    }
}

/// One-shot Gemma 4 pooled-embedding request under SlotAware (`Request::Embed`).
/// Embed has no decode loop, so it runs inline in the admit path (like
/// Warmup): reserve a scheduler slot for its KV, run the slot-aware embed at
/// that slot_id, then advance+release. Mirrors the legacy inflight Embed arm
/// (engine.rs:7839) — keeping the served-today capability, not 501-ing it.
#[allow(clippy::too_many_arguments)]
fn embed_gemma4_inline(
    guard: &mut Gemma4KvGuard<'_>,
    scheduler: &mut InflightBatchedScheduler,
    scheduler_stats_snapshot: &Arc<Mutex<SchedulerStats>>,
    per_slot_kv_budget_bytes: u64,
    kv_bytes_per_token: u64,
    prompt_tokens: Vec<u32>,
    reply: oneshot::Sender<Result<Vec<f32>>>,
) {
    // Embed has no decode budget; size the KV admit as prompt-only.
    let needed_bytes: u64 = if kv_bytes_per_token == 0 || per_slot_kv_budget_bytes == 0 {
        0
    } else {
        u64::from(prompt_tokens.len() as u32).saturating_mul(kv_bytes_per_token)
    };
    let admit_req = AdmitRequest {
        prompt_tokens: prompt_tokens.len() as u32,
        max_tokens: 1, // ≥1 so the scheduler allocates a physical slot
        // (max_tokens==0 → CompletedAtAdmit/no handle).
        kv_bytes_needed: needed_bytes,
    };
    let admitted = match scheduler.admit(admit_req) {
        Ok(slot) => slot,
        Err(e) => {
            let _ = reply.send(Err(anyhow::anyhow!(
                "ADR-040 Phase F M1 Embed admit failed: {e:?}"
            )));
            return;
        }
    };
    let Some(handle) = admitted.handle else {
        // No physical slot (shouldn't happen at max_tokens=1 with a free
        // slot). Fall back to the legacy single-seq embed at SlotId(0).
        let r = guard
            .model
            .weights
            .forward_embed_last(&prompt_tokens, &mut guard.model.ctx);
        let _ = reply.send(r);
        return;
    };
    clear_gemma4_self_mounts(guard.model);
    let result = embed_gemma4_slot_aware(
        guard.model,
        &prompt_tokens,
        &mut guard.kv,
        guard.hybrid.as_mut(),
        guard.dense.as_mut(),
        guard.mlx.as_mut(),
        handle.slot_id,
    );
    scheduler.advance_after_prefill(handle, prompt_tokens.len() as u32);
    reset_gemma4_slot(guard, handle.slot_id);
    scheduler.release(handle);
    if let Ok(mut g) = scheduler_stats_snapshot.lock() {
        *g = scheduler.stats();
    }
    let _ = reply.send(result);
}

/// ADR-040 §0.21 decode-category split — per-phase GPU-busy ns accumulators
/// (body = embed+30 layers; lm_head = final-norm+lm_head(m=N)+softcap). Populated
/// from `gpu_busy_ns()` deltas when `HF2Q_GPU_BUSY=1`; zero otherwise (the deltas
/// are 0 when the accumulator is off). Read+printed by the throughput probe.
static DECODE_BODY_GPU_NS: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0);
static DECODE_LMHEAD_GPU_NS: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0);

/// ADR-040 iter-I — vectorizable first-max argmax over a logits row, BYTE-IDENTICAL
/// to the scalar `v > bv` first-max loop (`argmax_f32_first_max_ref`): returns the
/// FIRST index of the maximum value and that value. Split into a max-reduction
/// then a first-equal scan; both auto-vectorize, unlike the loop-carried scalar
/// form (the N=8 decode critical path ran this 8× over a 256K vocab = 1.5ms/step
/// with the GPU idle). Edge parity with the scalar loop:
///   - NaN: skipped (Rust `f32::max` drops NaN; `v == maxv` is false for NaN).
///   - all -inf / all NaN / empty: returns (0, -inf) — index 0, value NEG_INFINITY.
#[inline]
fn argmax_f32_first_max(xs: &[f32]) -> (u32, f32) {
    let maxv = xs.iter().copied().fold(f32::NEG_INFINITY, f32::max);
    let bi = xs.iter().position(|&v| v == maxv).unwrap_or(0);
    (bi as u32, maxv)
}

/// Decode every active Gemma 4 slot in `handles` once (STEP 1: per-slot
/// forward in a loop). A slot that finishes this tick fires its reply and
/// is evicted; peers are untouched.
fn decode_batch_gemma4(
    guard: &mut Gemma4KvGuard<'_>,
    scheduler: &mut InflightBatchedScheduler,
    slots: &mut [Option<Gemma4Slot>],
    registration: Option<&super::registry::ModelRegistration>,
    handles: &[SlotHandle],
) {
    // First-max argmax (matches dispatch_argmax_f32's `>` tie-break). Only the
    // VALUE (== logits max) feeds the rerank threshold; the index is a cosmetic
    // seed that finalize_token_from_logits never lets affect the result (the max
    // is always within delta of itself ⇒ always a rerank candidate).
    // ADR-040 iter-I — vectorizable argmax (byte-identical to the scalar
    // first-max `v > bv` loop, but the two passes auto-vectorize where the
    // loop-carried-dependency scalar loop did not). See `argmax_f32_first_max`.
    fn argmax_f32(xs: &[f32]) -> (u32, f32) {
        argmax_f32_first_max(xs)
    }

    // ADR-040 S1c-2 — TWO-PASS batched-head tick. Pass 1 runs each live slot's
    // BODY only and gathers its final hidden row; ONE `lm_head_batched` then
    // amortizes the 605 MB lm_head weight read across all slots; Pass 2
    // finalizes each slot from its logits row. Per-slot output is bit-identical
    // to the prior per-slot full decode (H-S1-rowparity + shared
    // finalize_token_from_logits + decode_tick_finalize).
    let hs = guard.model.weights.hidden_size;
    let vocab = guard.model.weights.vocab_size;

    // ADR-040 Phase F `iter-F-prefill-determinism` (2026-06-24) — clear the
    // per-prefill self-mounts at the TOP of every decode tick. The slot-aware
    // decode uses a save-mount-RESTORE scope-guard on `self.{dense,hybrid,
    // leg_hb}_kv`; if a prefill-origin mount has crept onto these fields by any
    // path, each decode RESTORES it after its own forward, propagating the stale
    // mount across ticks until it poisons a later prefill's `is_none()` write-
    // back gate (forward_prefill.rs:970). The clear-after-prefill (admit) above
    // removed the admit-boundary entry (~13%→~3% on `slot_aware_staggered_
    // eviction`); this top-of-tick clear closes the residual cross-tick
    // propagation so the prior the scope-guard saves/restores is always None
    // (the clean-n4/n8 invariant). Data-lossless: per-slot K/V lives in the
    // persistent multi-seq scaffolds; decode re-mounts a fresh slice-view.
    let _hp_gather = std::time::Instant::now();
    clear_gemma4_self_mounts(guard.model);

    // ADR-040 iter-F-batched-determinism — env-gated per-tick trace
    // (HF2Q_DECODE_TRACE=1) to make the staggered batched non-determinism
    // observable: logs each tick's batch composition (N, per-slot pos+token)
    // and each slot's output token. Off by default (zero cost in production).
    let trace = std::env::var("HF2Q_DECODE_TRACE").is_ok();

    // Pass 1 — capture bodies. `captured` holds (handle, slot_idx, state, reply)
    // for each slot whose body ran; `hidden_rows` is their final hidden states
    // concatenated row-major `[n, hidden_size]`.
    // ADR-040 S2/S3: OPT-IN [N,hidden] batched body (HF2Q_BATCHED_BODY=1 +
    // hybrid KV). DEFAULT (flag off) = the proven per-slot capture path below.
    //
    // Phase F (2026-06-24) — a default AUTO-ENABLE at handles.len()>=2 was
    // attempted and REVERTED: although the batched body is byte-identical to
    // serial at N=1/4/8 (slot_aware_n1/n4/n8) AND ~1.94× faster at N=8 (198.8
    // ADR-040 `iter-F-batched-default` (2026-06-25) — DEFAULT-ON (opt out with
    // HF2Q_BATCHED_BODY=0). The prior non-determinism that blocked this flip was
    // root-caused in §0.16-RESOLVED: it was NOT the batched body — it was the
    // batched lm_head softcap covering only row 0 (softcap_params[1]=vocab, not
    // n*vocab), which affected BOTH decode paths. With that fixed, the batched
    // body is byte-identical to the serial slot-aware reference and to the
    // per-slot loop: validated by n1/n4/n8 parity, `staggered_eviction` 0/60, and
    // E2E long-generation coherence (8 concurrent distinct prompts × 600 tok over
    // HTTP, each byte-identical to its own serial ref, no cross-slot contamination
    // — at ~1.8× the serial aggregate throughput). The per-slot loop remains
    // available via the opt-out for A/B + as the byte-equiv-harness baseline.
    // ADR-040 M-SPEED-LC Stage 2 — batched body is eligible under EITHER
    // production KV regime now (see `BatchedKvRegime`): hybrid
    // (`guard.hybrid.is_some()`, HF2Q_HYBRID_KV=1 default) or full-TQ
    // (`guard.hybrid.is_none()`, HF2Q_HYBRID_KV=0 opt-in — the HB scaffold
    // `guard.kv` is unconditionally provisioned at spawn per H94, so it is
    // always available as the FullTq regime's backing buffers).
    let use_batched_body = std::env::var("HF2Q_BATCHED_BODY").as_deref() != Ok("0");
    let mut captured = Vec::new();
    let mut hidden_rows: Vec<f32> = Vec::new();
    // ADR-040 §25 iter-L — when HF2Q_FUSE_LMHEAD=1, the batched body encodes the
    // lm_head as the final CB-pipeline chunk and returns its output here (and an
    // empty hidden_rows); the head computation below uses this instead of a
    // separate lm_head_batched call (one commit_and_wait instead of two).
    let mut fused_head_out: Option<crate::inference::models::gemma4::batched_head::BatchedHeadOut> =
        None;
    if use_batched_body {
        let mut tokens: Vec<u32> = Vec::new();
        let mut sids: Vec<SlotId> = Vec::new();
        let mut positions: Vec<usize> = Vec::new();
        for &handle in handles {
            let slot_idx = handle.slot_id.0 as usize;
            let Some(slot) = slots.get_mut(slot_idx).and_then(Option::take) else {
                continue;
            };
            let (state, reply, installed) = slot;
            if installed != handle {
                slots[slot_idx] = Some((state, reply, installed));
                continue;
            }
            positions.push(state.prompt_len + state.generated_tokens.len() - 1);
            tokens.push(state.next_token);
            sids.push(handle.slot_id);
            captured.push((handle, slot_idx, state, reply));
        }
        if captured.is_empty() {
            return;
        }
        if trace {
            let comp: Vec<String> = sids
                .iter()
                .zip(positions.iter())
                .zip(tokens.iter())
                .map(|((s, p), t)| format!("s{}@{}:in{}", s.0, p, t))
                .collect();
            eprintln!("[DECTRACE] BATCHED N={} [{}]", sids.len(), comp.join(" "));
        }
        crate::inference::models::gemma4::batched_body::host_phases::add(
            crate::inference::models::gemma4::batched_body::host_phases::Phase::GatherMisc,
            _hp_gather.elapsed().as_nanos() as u64,
        );
        let _catsplit_g0 = std::time::Instant::now();
        // ADR-040 M-SPEED-LC Stage 2 (codex CHANGES-REQUIRED fix) — regime
        // selection is gated on `INVESTIGATION_ENV.hybrid_kv` (the SAME
        // HF2Q_HYBRID_KV-derived flag `provision_multi_seq_kv_for_slot_aware`
        // reads at spawn to decide whether to allocate `multi_seq_kv_hybrid`,
        // engine.rs:3242), NOT merely on `guard.hybrid.is_some()`. Selecting
        // FullTq whenever `guard.hybrid` happens to be `None` would silently
        // reroute onto the wrong scaffold if HF2Q_HYBRID_KV=1 (default) but
        // the hybrid scaffold failed to provision (a spawn-arm invariant
        // violation) — the older scalar slot-aware path treats that exact
        // condition as a typed `iter-C2c-cont-invariant-violated` error (see
        // `forward_prefill.rs:4587-4602`); this mirrors that, no silent
        // fallback.
        let body_res = if crate::debug::INVESTIGATION_ENV.hybrid_kv {
            match guard.hybrid.as_mut() {
                Some(hybrid) => {
                    let mut regime = crate::inference::models::gemma4::batched_body::BatchedKvRegime::Hybrid(
                        hybrid.as_mut_slice(),
                    );
                    let lm = &mut *guard.model;
                    lm.weights
                        .forward_decode_body_batched(
                            &tokens, &sids, &positions, &mut regime, &mut fused_head_out, &mut lm.ctx,
                        )
                }
                None => Err(anyhow::anyhow!(
                    "gemma4-batched-decode-hybrid-scaffold-absent (iter-C2c-cont-invariant-violated \
                     per ADR-040 §6.1.38 — HF2Q_HYBRID_KV=1 production-default; \
                     INVESTIGATION_ENV.hybrid_kv == true AT CALL TIME but guard.hybrid is None \
                     at the batched-body call site — iter-C2c-cont spawn-arm invariant violated \
                     (provision_multi_seq_kv_for_slot_aware must allocate MultiSeqHybridKvBuffers \
                     when HF2Q_HYBRID_KV=1; ADR-040 §6.1.33). ADR-040 M-SPEED-LC Stage 2 regime \
                     selection — no silent FullTq fallback."
                )),
            }
        } else {
            let mut regime =
                crate::inference::models::gemma4::batched_body::BatchedKvRegime::FullTq(
                    guard.kv.as_mut_slice(),
                );
            let lm = &mut *guard.model;
            lm.weights.forward_decode_body_batched(
                &tokens,
                &sids,
                &positions,
                &mut regime,
                &mut fused_head_out,
                &mut lm.ctx,
            )
        };
        DECODE_BODY_GPU_NS.fetch_add(
            _catsplit_g0.elapsed().as_nanos() as u64,
            std::sync::atomic::Ordering::Relaxed,
        );
        match body_res {
            Ok(h) => hidden_rows = h,
            Err(e) => {
                let msg = format!("{e}");
                for (handle, _slot_idx, _state, reply) in captured.drain(..) {
                    reset_gemma4_slot(guard, handle.slot_id);
                    scheduler.release(handle);
                    slot_fire_done(
                        reply,
                        Err(anyhow::anyhow!("forward_decode_body_batched: {msg}")),
                        false,
                    );
                }
                return;
            }
        }
    } else {
        for &handle in handles {
            let slot_idx = handle.slot_id.0 as usize;
            let Some(slot) = slots.get_mut(slot_idx).and_then(Option::take) else {
                continue;
            };
            let (mut state, reply, installed) = slot;
            if installed != handle {
                // Stale handle for this physical slot: put it back untouched.
                slots[slot_idx] = Some((state, reply, installed));
                continue;
            }
            match state.decode_tick_capture(
                guard.model,
                &mut guard.kv,
                guard.hybrid.as_mut(),
                guard.dense.as_mut(),
                guard.mlx.as_mut(),
            ) {
                Ok(hidden) => {
                    hidden_rows.extend_from_slice(&hidden);
                    captured.push((handle, slot_idx, state, reply));
                }
                Err(e) => {
                    // Body forward error: evict this slot with a typed error.
                    reset_gemma4_slot(guard, handle.slot_id);
                    scheduler.release(handle);
                    slot_fire_done(reply, Err(e), false);
                }
            }
        }
        if captured.is_empty() {
            return;
        }
    }
    let n = captured.len();

    // Batched head: ONE final-norm + lm_head(m=N) + softcap for all slots.
    // §25 iter-L: when HF2Q_FUSE_LMHEAD=1, the body already encoded the head into
    // its CB pipeline and produced `fused_head_out` (its GPU time folds into
    // DECODE_BODY_GPU_NS, so DECODE_LMHEAD_GPU_NS reads ~0 on the fused path).
    let _catsplit_h0 = std::time::Instant::now();
    let head = if let Some(h) = fused_head_out.take() {
        h
    } else {
        match guard
            .model
            .weights
            .lm_head_batched(&hidden_rows, n, &mut guard.model.ctx)
        {
            Ok(h) => h,
            Err(e) => {
                // Head failure is fatal for this tick's slots (no tokens
                // producible); evict each. anyhow::Error isn't Clone, carry msg.
                let msg = format!("{e}");
                for (handle, _slot_idx, _state, reply) in captured {
                    reset_gemma4_slot(guard, handle.slot_id);
                    scheduler.release(handle);
                    slot_fire_done(reply, Err(anyhow::anyhow!("lm_head_batched: {msg}")), false);
                }
                return;
            }
        }
    };

    DECODE_LMHEAD_GPU_NS.fetch_add(
        _catsplit_h0.elapsed().as_nanos() as u64,
        std::sync::atomic::Ordering::Relaxed,
    );

    // Pass 2 — finalize each slot from its logits row.
    let _hp_sample = std::time::Instant::now();
    for (i, (handle, slot_idx, mut state, reply)) in captured.into_iter().enumerate() {
        let logits_row = &head.logits[i * vocab..(i + 1) * vocab];
        let normed_row = &head.normed[i * hs..(i + 1) * hs];
        // Greedy token: the GPU-argmax index is irrelevant to the rerank and the
        // top1 VALUE (== CPU max) is bit-identical, so a CPU argmax reproduces
        // the scalar head's greedy token exactly.
        let _hp_am = std::time::Instant::now();
        // ADR-040 §26 iter-M: use GPU-side argmax+candidate set (drops the ~0.92ms
        // host full-vocab scan) when available + not overflowed; the cheap F64
        // rerank still runs on host. BYTE-IDENTICAL: GPU candidate set == host
        // threshold scan, both feed the same rerank tail. Host fallback on
        // overflow (rare) or HF2Q_GPU_SAMPLE off.
        let gpu_s = head
            .gpu_sample
            .as_ref()
            .filter(|gs| gs.overflow[i] == 0 && (gs.cand_count[i] as usize) <= gs.cap);
        let top1_val: f32;
        let greedy_result = if let Some(gs) = gpu_s {
            top1_val = gs.top1_val[i];
            let cnt = (gs.cand_count[i] as usize).min(gs.cap);
            let cands = &gs.cand_ids[i * gs.cap..i * gs.cap + cnt];
            guard.model.weights.finalize_token_from_gpu_candidates(
                cands,
                normed_row,
                gs.top1_idx[i],
            )
        } else {
            let (ti, tv) = argmax_f32(logits_row);
            top1_val = tv;
            guard
                .model
                .weights
                .finalize_token_from_logits(logits_row, normed_row, ti, tv)
        };
        let greedy_token = match greedy_result {
            Ok(t) => t,
            Err(e) => {
                reset_gemma4_slot(guard, handle.slot_id);
                scheduler.release(handle);
                slot_fire_done(reply, Err(e), false);
                continue;
            }
        };
        crate::inference::models::gemma4::batched_body::host_phases::add(
            crate::inference::models::gemma4::batched_body::host_phases::Phase::ArgmaxFinalize,
            _hp_am.elapsed().as_nanos() as u64,
        );
        if trace {
            eprintln!(
                "[DECTRACE]   out s{} top1_val={:.4} -> tok{}",
                handle.slot_id.0, top1_val, greedy_token
            );
        }
        let _hp_dt = std::time::Instant::now();
        let tick = match state.decode_tick_finalize(guard.model, greedy_token, logits_row) {
            Ok(t) => t,
            Err(e) => {
                reset_gemma4_slot(guard, handle.slot_id);
                scheduler.release(handle);
                slot_fire_done(reply, Err(e), false);
                continue;
            }
        };
        crate::inference::models::gemma4::batched_body::host_phases::add(
            crate::inference::models::gemma4::batched_body::host_phases::Phase::DecodeTick,
            _hp_dt.elapsed().as_nanos() as u64,
        );

        let client_dropped = slot_emit_token(&reply, &tick);
        scheduler.advance_after_decode(handle);

        if tick.finished || client_dropped {
            reset_gemma4_slot(guard, handle.slot_id);
            scheduler.release(handle);
            let gr = Ok(state.finish(registration));
            slot_fire_done(reply, gr, client_dropped);
        } else {
            // Still generating: re-seat the slot for the next tick.
            slots[slot_idx] = Some((state, reply, handle));
        }
    }
    crate::inference::models::gemma4::batched_body::host_phases::add(
        crate::inference::models::gemma4::batched_body::host_phases::Phase::SampleLoop,
        _hp_sample.elapsed().as_nanos() as u64,
    );
}

/// Per-slot KV exit reset for Gemma 4 (mirror of the serial ref's exit
/// reset). Swallows errors (logged) — a failed reset is observability-only
/// because the next admission's entry reset re-cleans the slot.
/// Clear the gemma4 forward's per-prefill self-mounted KV scratch
/// (`self.dense_kvs` / `self.hybrid_kv` / `self.leg_hb_encoded`) back to the
/// `None` fresh state the slot-aware prefill consume-gate expects on entry
/// (forward_prefill.rs:699 "SerialFifo enters with self.dense_kvs == None").
///
/// The slot-aware prefill mounts a per-slot slice-VIEW into these shared
/// `MlxModelWeights` fields and leaves them `Some` at the end (the legacy
/// single-seq write-back, forward_prefill.rs:2344/3663/3865). Under F1's
/// N>1 interleave, slot A's leftover view poisons slot B's prefill: B's
/// consume-gate sees A's view with A's capacity and bails
/// (`capacity X < required Y`). The actual per-slot KV is safe in the
/// per-slot `multi_seq_kv*` scaffold (the views aliased it), so dropping
/// these Options loses nothing. This MIRRORS the codebase's own precedent
/// in `forward_embed_last` (forward_prefill.rs:2459-2461), which clears the
/// same fields before re-entering prefill precisely to avoid the
/// "first allocation's capacity poisons every subsequent call" fault.
///
/// hf2q-side, worker-owned, no forward-path change: it restores a
/// documented precondition between slots, it does not alter the forward.
fn clear_gemma4_self_mounts(g: &mut GemmaLoadedModel) {
    g.weights.dense_kvs = None;
    g.weights.hybrid_kv = None;
    g.weights.leg_hb_encoded = None;
}

fn reset_gemma4_slot(guard: &mut Gemma4KvGuard<'_>, slot_id: SlotId) {
    for (layer_idx, buf) in guard.kv.iter_mut().enumerate() {
        if let Err(e) = buf.reset_for_slot(slot_id) {
            tracing::warn!(
                "Gemma4 SlotAware exit reset L{layer_idx} slot={} failed: {e}",
                slot_id.0
            );
        }
    }
    if let Some(hybrid) = guard.hybrid.as_mut() {
        for (layer_idx, buf) in hybrid.iter_mut().enumerate() {
            if let Err(e) = buf.reset_for_slot(slot_id) {
                tracing::warn!(
                    "Gemma4 SlotAware exit reset (hybrid) L{layer_idx} slot={} failed: {e}",
                    slot_id.0
                );
            }
        }
    }
}

/// Answer every pending request with a typed startup error so callers do
/// not hang when a SlotAware loop cannot start (e.g. KV scaffold absent).
fn drain_with_startup_error(mut rx: mpsc::Receiver<Request>, why: &str) {
    while let Some(req) = rx.blocking_recv() {
        match req {
            Request::Generate { reply, .. } => {
                let _ = reply.send(Err(anyhow::anyhow!(
                    "capability_unsupported: ADR-040 Phase F M1 SlotAware loop \
                     could not start: {why}"
                )));
            }
            Request::GenerateStream { events, .. } => {
                let _ = events.blocking_send(super::sse::GenerationEvent::Error(format!(
                    "capability_unsupported: ADR-040 Phase F M1 SlotAware loop \
                     could not start: {why}"
                )));
            }
            Request::Warmup { reply } => {
                let _ = reply.send(Ok(()));
            }
            Request::Embed { reply, .. } => {
                let _ = reply.send(Err(anyhow::anyhow!(
                    "capability_unsupported: ADR-040 Phase F M1 SlotAware loop \
                     could not start: {why}"
                )));
            }
            Request::GenerateWithSoftTokens { reply, .. } => {
                let _ = reply.send(Err(anyhow::anyhow!(
                    "capability_unsupported: ADR-040 Phase F M1 SlotAware loop \
                     could not start: {why}"
                )));
            }
            Request::Shutdown => break,
            _ => {}
        }
    }
}

// ---------------------------------------------------------------------------
// Qwen35 SlotAware loop (F1). Lives here (not engine_qwen35.rs) because the
// loop drives the engine-private `Request` enum + scheduler + shared reply
// helpers; the per-arch decode SEAM (`Qwen35DecodeState`) lives beside its
// serial reference in engine_qwen35.rs.
// ---------------------------------------------------------------------------

/// Scope guard holding the Qwen35 persistent `HybridKvCache` taken out of
/// the model for the SlotAware loop lifetime; restores on every exit path
/// via `Drop` (ADR-040 Phase F M1 lead requirement). Unlike Gemma 4's
/// per-layer `Vec`, Qwen35 owns a single `HybridKvCache` with an `n_seqs`
/// slot dimension.
struct Qwen35KvGuard<'a> {
    model: &'a mut super::engine_qwen35::Qwen35LoadedModel,
    /// `Some` for the loop lifetime; `Drop` `take()`s it back into the
    /// model. Stored as `Option` because `HybridKvCache` is move-only (no
    /// `Default`) so there is no placeholder to `mem::take` against.
    kv: Option<crate::inference::models::qwen35::kv_cache::HybridKvCache>,
}

impl<'a> Qwen35KvGuard<'a> {
    fn take(model: &'a mut super::engine_qwen35::Qwen35LoadedModel) -> Result<Self> {
        let kv = model.persistent_kv_cache.take().ok_or_else(|| {
            anyhow::anyhow!(
                "capability_unsupported: ADR-040 Phase F M1 — persistent_kv_cache is None \
                 for Qwen35 SlotAware loop entry. C2d spawn-arm invariant violated."
            )
        })?;
        Ok(Self {
            model,
            kv: Some(kv),
        })
    }
}

impl Drop for Qwen35KvGuard<'_> {
    fn drop(&mut self) {
        self.model.persistent_kv_cache = self.kv.take();
    }
}

type Qwen35Slot = (
    super::engine_qwen35::Qwen35DecodeState,
    SlotReply,
    SlotHandle,
);

/// ADR-040 Phase F M1 (F1) — Qwen35 SlotAware admit-while-decoding loop.
/// Same shape as `run_slot_aware_gemma4`; single shared `HybridKvCache`.
#[allow(clippy::too_many_arguments)]
fn run_slot_aware_qwen35(
    mut model: super::engine_qwen35::Qwen35LoadedModel,
    mut rx: mpsc::Receiver<Request>,
    registration: Option<super::registry::ModelRegistration>,
    mut scheduler: InflightBatchedScheduler,
    scheduler_stats_snapshot: Arc<Mutex<SchedulerStats>>,
    per_slot_kv_budget_bytes: u64,
    kv_bytes_per_token: u64,
) {
    let mut guard = match Qwen35KvGuard::take(&mut model) {
        Ok(g) => g,
        Err(e) => {
            tracing::error!("Qwen35 SlotAware loop cannot start: {e:#}");
            drain_with_startup_error(rx, "Qwen35 persistent_kv_cache absent at SlotAware entry");
            return;
        }
    };
    let n_slots = guard.kv.as_ref().expect("kv Some at entry").n_seqs as usize;
    let mut slots: Vec<Option<Qwen35Slot>> = (0..n_slots).map(|_| None).collect();

    let publish = |sched: &InflightBatchedScheduler, snap: &Arc<Mutex<SchedulerStats>>| {
        if let Ok(mut g) = snap.lock() {
            *g = sched.stats();
        }
    };
    let mut pending: Option<Request> = None;

    'worker: loop {
        // ── ADMIT ────────────────────────────────────────────────────
        loop {
            let Some(_free) = slots.iter().position(|s| s.is_none()) else {
                break;
            };
            let req = match pending.take() {
                Some(r) => r,
                None => match rx.try_recv() {
                    Ok(r) => r,
                    Err(mpsc::error::TryRecvError::Empty) => break,
                    Err(mpsc::error::TryRecvError::Disconnected) => break 'worker,
                },
            };
            match req {
                Request::Generate {
                    prompt_tokens,
                    params,
                    reply,
                } => {
                    admit_qwen35_slot(
                        &mut guard,
                        &mut scheduler,
                        &mut slots,
                        registration.as_ref(),
                        &scheduler_stats_snapshot,
                        per_slot_kv_budget_bytes,
                        kv_bytes_per_token,
                        prompt_tokens,
                        params,
                        SlotReply::Unary(reply),
                    );
                    publish(&scheduler, &scheduler_stats_snapshot);
                }
                Request::GenerateStream {
                    prompt_tokens,
                    params,
                    events,
                    cancellation_counter,
                    ..
                } => {
                    admit_qwen35_slot(
                        &mut guard,
                        &mut scheduler,
                        &mut slots,
                        registration.as_ref(),
                        &scheduler_stats_snapshot,
                        per_slot_kv_budget_bytes,
                        kv_bytes_per_token,
                        prompt_tokens,
                        params,
                        SlotReply::Stream {
                            events,
                            cancel: cancellation_counter,
                        },
                    );
                    publish(&scheduler, &scheduler_stats_snapshot);
                }
                Request::Warmup { reply } => {
                    // Qwen35 warmup is a no-op (mirror of the SerialFifo arm).
                    let _ = reply.send(Ok(()));
                }
                Request::Shutdown => {
                    tracing::info!("Qwen35 SlotAware worker received Shutdown; exiting");
                    break 'worker;
                }
                // Embed IS served at SlotId>0 by the legacy inflight arm
                // today (embed_qwen35_slot_aware, engine_qwen35.rs:5288) →
                // 501 would regress. One-shot, run inline in admit.
                Request::Embed {
                    prompt_tokens,
                    reply,
                } => {
                    embed_qwen35_inline(
                        &mut guard,
                        &mut scheduler,
                        &scheduler_stats_snapshot,
                        per_slot_kv_budget_bytes,
                        kv_bytes_per_token,
                        prompt_tokens,
                        reply,
                    );
                    publish(&scheduler, &scheduler_stats_snapshot);
                }
                // GenerateWithSoftTokens IS served at SlotId>0 today
                // (generate_qwen35_once_with_soft_tokens[_and_deepstack]_slot_aware,
                // engine_qwen35.rs:5460/5783) → 501 would regress. The
                // qwen35 soft-token forward path carries deepstack +
                // 3D-mRoPE positions (Wedge-4) with a distinct forward
                // primitive, so — unlike the gemma4 shared-prefill path —
                // it runs inline in admit via the existing slot-aware
                // soft-token generator (which does its own full decode),
                // reusing the same primitive the legacy arm called. This
                // preserves the capability exactly; STEP-2/F2 can fold it
                // into the batched loop once the soft-token decode primitive
                // is threaded through the per-slot scaffold.
                Request::GenerateWithSoftTokens {
                    prompt_tokens,
                    soft_tokens,
                    params,
                    deepstack,
                    positions_flat,
                    reply,
                } => {
                    generate_qwen35_soft_tokens_inline(
                        &mut guard,
                        &mut scheduler,
                        registration.as_ref(),
                        &scheduler_stats_snapshot,
                        per_slot_kv_budget_bytes,
                        kv_bytes_per_token,
                        prompt_tokens,
                        soft_tokens,
                        deepstack,
                        positions_flat,
                        params,
                        reply,
                    );
                    publish(&scheduler, &scheduler_stats_snapshot);
                }
                _ => {}
            }
        }

        // ── STEP ─────────────────────────────────────────────────────
        let step = match scheduler.step() {
            Ok(s) => s,
            Err(e) => {
                tracing::error!("Qwen35 SlotAware scheduler.step() failed: {e:?}");
                break 'worker;
            }
        };
        match step {
            SchedulerStep::Idle => match rx.blocking_recv() {
                Some(r) => pending = Some(r),
                None => break 'worker,
            },
            SchedulerStep::Prefill { .. } => {}
            SchedulerStep::Decode { handles } => {
                decode_batch_qwen35(
                    &mut guard,
                    &mut scheduler,
                    &mut slots,
                    registration.as_ref(),
                    &handles,
                );
                let _hp_pub = std::time::Instant::now();
                publish(&scheduler, &scheduler_stats_snapshot);
                crate::inference::models::gemma4::batched_body::host_phases::add(
                    crate::inference::models::gemma4::batched_body::host_phases::Phase::Publish,
                    _hp_pub.elapsed().as_nanos() as u64,
                );
            }
            SchedulerStep::Mixed { decode_handles, .. } => {
                decode_batch_qwen35(
                    &mut guard,
                    &mut scheduler,
                    &mut slots,
                    registration.as_ref(),
                    &decode_handles,
                );
                let _hp_pub = std::time::Instant::now();
                publish(&scheduler, &scheduler_stats_snapshot);
                crate::inference::models::gemma4::batched_body::host_phases::add(
                    crate::inference::models::gemma4::batched_body::host_phases::Phase::Publish,
                    _hp_pub.elapsed().as_nanos() as u64,
                );
            }
        }
    }

    drop(guard);
    tracing::info!("Qwen35 SlotAware worker thread exited");
}

#[allow(clippy::too_many_arguments)]
fn admit_qwen35_slot(
    guard: &mut Qwen35KvGuard<'_>,
    scheduler: &mut InflightBatchedScheduler,
    slots: &mut [Option<Qwen35Slot>],
    registration: Option<&super::registry::ModelRegistration>,
    scheduler_stats_snapshot: &Arc<Mutex<SchedulerStats>>,
    per_slot_kv_budget_bytes: u64,
    kv_bytes_per_token: u64,
    prompt_tokens: Vec<u32>,
    params: SamplingParams,
    reply: SlotReply,
) {
    let needed_bytes: u64 = if kv_bytes_per_token == 0 || per_slot_kv_budget_bytes == 0 {
        0
    } else {
        u64::from(prompt_tokens.len() as u32)
            .saturating_add(u64::from(params.max_tokens as u32))
            .saturating_mul(kv_bytes_per_token)
    };
    let admit_req = AdmitRequest {
        prompt_tokens: prompt_tokens.len() as u32,
        max_tokens: params.max_tokens as u32,
        kv_bytes_needed: needed_bytes,
    };
    let admitted = match scheduler.admit(admit_req) {
        Ok(slot) => slot,
        Err(AdmitError::SlotBudgetExceeded {
            needed_bytes,
            budget_bytes,
        }) => {
            slot_fire_done(
                reply,
                Err(anyhow::anyhow!(
                    "slot_budget_exceeded: ADR-040 Phase F M1 — per-slot KV budget \
                     exceeded (needed_bytes={needed_bytes}, budget_bytes={budget_bytes})."
                )),
                false,
            );
            return;
        }
        Err(e) => {
            slot_fire_done(
                reply,
                Err(anyhow::anyhow!("ADR-040 Phase F M1 admit failed: {e:?}")),
                false,
            );
            return;
        }
    };
    let Some(handle) = admitted.handle else {
        // CompletedAtAdmit (`max_tokens == 0`): mirror the SerialFifo arm's
        // fallback — run the legacy non-slot-aware `generate_qwen35_once`
        // and fire, no scheduler bookkeeping (byte-equivalence).
        let gr = super::engine_qwen35::generate_qwen35_once(
            guard.model,
            &prompt_tokens,
            &params,
            registration,
        );
        slot_fire_done(reply, gr, false);
        return;
    };

    let seed = super::engine_qwen35::Qwen35DecodeState::prefill_seed(
        guard.model,
        &prompt_tokens,
        &params,
        guard.kv.as_mut().expect("kv Some during loop"),
        handle.slot_id,
    );
    let state = match seed {
        Ok(s) => s,
        Err(e) => {
            let _ = guard
                .kv
                .as_mut()
                .expect("kv Some during loop")
                .reset_for_slot(handle.slot_id);
            scheduler.release(handle);
            if let Ok(mut g) = scheduler_stats_snapshot.lock() {
                *g = scheduler.stats();
            }
            slot_fire_done(reply, Err(e), false);
            return;
        }
    };

    scheduler.advance_after_prefill(handle, prompt_tokens.len() as u32);

    if state.finished_at_seed() {
        // Stream: emit the seed text. finish() owns the assembled result.
        let tick = TickOutcome {
            fragment: state.seed_fragment(),
            is_reasoning: false,
            finished: true,
        };
        let dropped = slot_emit_token(&reply, &tick);
        scheduler.advance_after_decode(handle);
        let _ = guard
            .kv
            .as_mut()
            .expect("kv Some during loop")
            .reset_for_slot(handle.slot_id);
        scheduler.release(handle);
        let gr = Ok(state.finish(guard.model, registration));
        slot_fire_done(reply, gr, dropped);
        return;
    }

    let slot_idx = handle.slot_id.0 as usize;
    slots[slot_idx] = Some((state, reply, handle));
}

/// One-shot Qwen35 pooled-embedding (`Request::Embed`) under SlotAware —
/// inline in admit (no decode loop), mirroring the legacy inflight arm
/// (embed_qwen35_slot_aware, engine_qwen35.rs:5288). Keeps the served-today
/// capability rather than 501-ing it.
fn embed_qwen35_inline(
    guard: &mut Qwen35KvGuard<'_>,
    scheduler: &mut InflightBatchedScheduler,
    scheduler_stats_snapshot: &Arc<Mutex<SchedulerStats>>,
    per_slot_kv_budget_bytes: u64,
    kv_bytes_per_token: u64,
    prompt_tokens: Vec<u32>,
    reply: oneshot::Sender<Result<Vec<f32>>>,
) {
    let needed_bytes: u64 = if kv_bytes_per_token == 0 || per_slot_kv_budget_bytes == 0 {
        0
    } else {
        u64::from(prompt_tokens.len() as u32).saturating_mul(kv_bytes_per_token)
    };
    let admit_req = AdmitRequest {
        prompt_tokens: prompt_tokens.len() as u32,
        max_tokens: 1,
        kv_bytes_needed: needed_bytes,
    };
    let admitted = match scheduler.admit(admit_req) {
        Ok(slot) => slot,
        Err(e) => {
            let _ = reply.send(Err(anyhow::anyhow!(
                "ADR-040 Phase F M1 Qwen35 Embed admit failed: {e:?}"
            )));
            return;
        }
    };
    let Some(handle) = admitted.handle else {
        let _ = reply.send(Err(anyhow::anyhow!(
            "ADR-040 Phase F M1 — Qwen35 Embed got no physical slot (capacity \
             mismatch)."
        )));
        return;
    };
    let result = super::engine_qwen35::embed_qwen35_slot_aware(
        guard.model,
        &prompt_tokens,
        guard.kv.as_mut().expect("kv Some during loop"),
        handle.slot_id,
    );
    scheduler.advance_after_prefill(handle, prompt_tokens.len() as u32);
    let _ = guard
        .kv
        .as_mut()
        .expect("kv Some during loop")
        .reset_for_slot(handle.slot_id);
    scheduler.release(handle);
    if let Ok(mut g) = scheduler_stats_snapshot.lock() {
        *g = scheduler.stats();
    }
    let _ = reply.send(result);
}

/// Qwen35 `Request::GenerateWithSoftTokens` under SlotAware — run inline in
/// admit via the existing slot-aware soft-token generator (which carries
/// deepstack + 3D-mRoPE positions and does its own full decode). Reuses the
/// SAME primitive the legacy inflight arm called (engine.rs:8548/8560), so
/// it preserves the served-today capability exactly; folding it into the
/// batched decode loop is STEP-2/F2 work (the soft-token decode primitive
/// must be threaded through the per-slot scaffold first).
#[allow(clippy::too_many_arguments)]
fn generate_qwen35_soft_tokens_inline(
    guard: &mut Qwen35KvGuard<'_>,
    scheduler: &mut InflightBatchedScheduler,
    registration: Option<&super::registry::ModelRegistration>,
    scheduler_stats_snapshot: &Arc<Mutex<SchedulerStats>>,
    per_slot_kv_budget_bytes: u64,
    kv_bytes_per_token: u64,
    prompt_tokens: Vec<u32>,
    soft_tokens: Vec<SoftTokenData>,
    deepstack: Option<DeepstackData>,
    positions_flat: Option<Vec<i32>>,
    params: SamplingParams,
    reply: oneshot::Sender<Result<GenerationResult>>,
) {
    let needed_bytes: u64 = if kv_bytes_per_token == 0 || per_slot_kv_budget_bytes == 0 {
        0
    } else {
        u64::from(prompt_tokens.len() as u32)
            .saturating_add(u64::from(params.max_tokens as u32))
            .saturating_mul(kv_bytes_per_token)
    };
    let admit_req = AdmitRequest {
        prompt_tokens: prompt_tokens.len() as u32,
        max_tokens: params.max_tokens as u32,
        kv_bytes_needed: needed_bytes,
    };
    let admitted = match scheduler.admit(admit_req) {
        Ok(slot) => slot,
        Err(e) => {
            let _ = reply.send(Err(anyhow::anyhow!(
                "ADR-040 Phase F M1 Qwen35 SoftTokens admit failed: {e:?}"
            )));
            return;
        }
    };
    // Build borrowed injections + deepstack from owned data (mirror of the
    // legacy arm engine.rs:8530-8542).
    let injections: Vec<crate::serve::forward_prefill::SoftTokenInjection<'_>> = soft_tokens
        .iter()
        .map(|d| crate::serve::forward_prefill::SoftTokenInjection {
            range: d.range.clone(),
            embeddings: &d.embeddings,
        })
        .collect();
    let ds_borrow: Option<crate::serve::forward_prefill::DeepstackInjection<'_>> = deepstack
        .as_ref()
        .map(|d| crate::serve::forward_prefill::DeepstackInjection {
            image_token_positions: d.image_token_positions.clone(),
            chunks: d.chunks.iter().collect(),
        });
    let Some(handle) = admitted.handle else {
        // max_tokens==0 CompletedAtAdmit: no slot. Fall back to the legacy
        // non-slot-aware soft-token generator at SlotId(0).
        let result = if ds_borrow.is_some() || positions_flat.is_some() {
            super::engine_qwen35::generate_qwen35_once_with_soft_tokens_and_deepstack(
                guard.model,
                &prompt_tokens,
                &injections,
                ds_borrow.as_ref(),
                positions_flat.as_deref(),
                &params,
                registration,
            )
        } else {
            super::engine_qwen35::generate_qwen35_once_with_soft_tokens(
                guard.model,
                &prompt_tokens,
                &injections,
                &params,
                registration,
            )
        };
        let _ = reply.send(result);
        return;
    };
    let kv = guard.kv.as_mut().expect("kv Some during loop");
    let result = if ds_borrow.is_some() || positions_flat.is_some() {
        super::engine_qwen35::generate_qwen35_once_with_soft_tokens_and_deepstack_slot_aware(
            guard.model,
            &prompt_tokens,
            &injections,
            ds_borrow.as_ref(),
            positions_flat.as_deref(),
            &params,
            registration,
            kv,
            handle.slot_id,
        )
    } else {
        super::engine_qwen35::generate_qwen35_once_with_soft_tokens_slot_aware(
            guard.model,
            &prompt_tokens,
            &injections,
            &params,
            registration,
            kv,
            handle.slot_id,
        )
    };
    scheduler.advance_after_prefill(handle, prompt_tokens.len() as u32);
    if let Ok(ref gr) = result {
        for _ in 0..gr.completion_tokens {
            scheduler.advance_after_decode(handle);
        }
    }
    let _ = guard
        .kv
        .as_mut()
        .expect("kv Some during loop")
        .reset_for_slot(handle.slot_id);
    scheduler.release(handle);
    if let Ok(mut g) = scheduler_stats_snapshot.lock() {
        *g = scheduler.stats();
    }
    let _ = reply.send(result);
}

fn decode_batch_qwen35(
    guard: &mut Qwen35KvGuard<'_>,
    scheduler: &mut InflightBatchedScheduler,
    slots: &mut [Option<Qwen35Slot>],
    registration: Option<&super::registry::ModelRegistration>,
    handles: &[SlotHandle],
) {
    for &handle in handles {
        let slot_idx = handle.slot_id.0 as usize;
        let Some(slot) = slots.get_mut(slot_idx).and_then(Option::take) else {
            continue;
        };
        let (mut state, reply, installed) = slot;
        if installed != handle {
            slots[slot_idx] = Some((state, reply, installed));
            continue;
        }

        let qtick =
            match state.decode_tick(guard.model, guard.kv.as_mut().expect("kv Some during loop")) {
                Ok(t) => t,
                Err(e) => {
                    let _ = guard
                        .kv
                        .as_mut()
                        .expect("kv Some during loop")
                        .reset_for_slot(handle.slot_id);
                    scheduler.release(handle);
                    slot_fire_done(reply, Err(e), false);
                    continue;
                }
            };
        let tick = TickOutcome {
            fragment: qtick.fragment,
            is_reasoning: qtick.is_reasoning,
            finished: qtick.finished,
        };

        let client_dropped = slot_emit_token(&reply, &tick);
        scheduler.advance_after_decode(handle);

        if tick.finished || client_dropped {
            let _ = guard
                .kv
                .as_mut()
                .expect("kv Some during loop")
                .reset_for_slot(handle.slot_id);
            scheduler.release(handle);
            let gr = Ok(state.finish(guard.model, registration));
            slot_fire_done(reply, gr, client_dropped);
        } else {
            slots[slot_idx] = Some((state, reply, handle));
        }
    }
}

fn worker_run(
    mut loaded: LoadedModel,
    mut rx: mpsc::Receiver<Request>,
    registration: Option<super::registry::ModelRegistration>,
    mode: EngineMode,
    queue_capacity: u32,
    scheduler_stats_snapshot: Arc<Mutex<SchedulerStats>>,
    per_slot_kv_budget_bytes: u64,
    kv_bytes_per_token: u64,
) {
    tracing::info!(
        model = %loaded.model_id(),
        ?mode,
        queue_capacity,
        per_slot_kv_budget_bytes,
        kv_bytes_per_token,
        "hf2q-engine worker thread started"
    );

    // ADR-040 Phase F M1 (F1) — SlotAware now runs a SEPARATE
    // scheduler-driven, admit-while-decoding worker loop
    // (`worker_run_slot_aware`) that drives `scheduler.step()` and
    // interleaves up to `max_slots` concurrent requests across decode
    // ticks. Dispatching here — BEFORE the SerialFifo scheduler is even
    // constructed — keeps the legacy `blocking_recv` drain below
    // byte-identical for SerialFifo (the production default): zero edits
    // to that body, so `engine_serial_fifo_byte_equivalent_to_pre_phase_c`
    // holds by construction. The inlined SlotId(N>0) arms further down in
    // the SerialFifo body become dead for SerialFifo (it only ever sees
    // SlotId(0) from the FIFO adapter); they are retained as-is for the
    // SerialFifo path's existing source-introspection pins and removed as
    // M1-completion hygiene, not here.
    if let EngineMode::SlotAware { max_slots } = mode {
        worker_run_slot_aware(
            loaded,
            rx,
            registration,
            max_slots,
            queue_capacity,
            scheduler_stats_snapshot,
            per_slot_kv_budget_bytes,
            kv_bytes_per_token,
        );
        return;
    }

    // ADR-040 Phase C iter-2a (C2b) — construct the scheduler at thread
    // entry per dossier §2.3 + §4 iter-2a step 3. Under Shape A the
    // scheduler is owned exclusively by the worker thread (no `Send +
    // Sync` Arc<Mutex<...>> contention). The `SlotAware` arm is
    // genuinely unreachable here because `Engine::spawn_with_mode`
    // (iter-1.5 F1, engine.rs:2636-2667) rejects it with
    // `EngineSpawnError::ModeNotYetWired` BEFORE the worker thread is
    // spawned — the `unreachable!` macro is the defensive surface that
    // surfaces any future caller that bypasses the spawn-time
    // rejection. Per ADR-040 §7 the macro is permissible in
    // genuinely-unreachable branches (matches existing codebase
    // patterns); typed sentinel errors handle the operator-actionable
    // surface, `unreachable!` handles the compile-time-impossible
    // surface.
    //
    // Per dossier §2.9 / §2.8: the `Scheduler` trait surface
    // (`policy`/`admit`/`step`/`release`/`stats`) deliberately does NOT
    // include `advance_after_prefill` / `advance_after_decode` — those
    // callbacks live on the concrete `FifoSchedulerAdapter` /
    // `InflightBatchedScheduler` types because their FSM-advance
    // surface differs (FIFO has no chunking). Under Shape A iter-2a
    // we hold the concrete adapter directly (not boxed) so the
    // advance callbacks are accessible without dynamic dispatch or
    // downcasting. The dossier §4 iter-2a step 3 snippet wrote
    // `Box<dyn Scheduler>` for narrative consistency; the concrete-
    // adapter shape used here is the implementation realisation that
    // honors §2.9's "advance lives on concrete type" pin.
    // ADR-040 §3.5 iter-A5b — scheduler-side per-slot KV budget
    // wiring.  `per_slot_kv_budget_bytes == 0` means "enforcement
    // disabled" (preserves pre-A5 byte-equivalence for operators who
    // do not set `--kv-cache-budget-bytes`).  The wrap helper
    // `new_with_kv_budget` accepts `0` and is byte-equivalent to
    // `new(queue_capacity)` in that case (per scheduler.rs tests).
    // ADR-040 Phase C iter-2c (C2c): switch from concrete
    // `FifoSchedulerAdapter` to the `WorkerScheduler` enum so the
    // SlotAware arm can construct `InflightBatchedScheduler` without
    // boxing (per dossier §2.9 the advance APIs live on the concrete
    // type — `Box<dyn Scheduler>` would lose access). FifoSerial
    // continues to construct exactly the same `FifoSchedulerAdapter`
    // (byte-equivalence H23 pin); the only addition is the
    // `WorkerScheduler::Inflight` arm for SlotAware.
    let mut scheduler: WorkerScheduler = match mode {
        EngineMode::SerialFifo => WorkerScheduler::Fifo(FifoSchedulerAdapter::new_with_kv_budget(
            queue_capacity,
            per_slot_kv_budget_bytes,
        )),
        EngineMode::SlotAware { max_slots } => {
            WorkerScheduler::Inflight(InflightBatchedScheduler::new_with_kv_budget(
                queue_capacity,
                max_slots,
                per_slot_kv_budget_bytes,
            ))
        }
    };

    // Helper closure: push the current SchedulerStats snapshot to the
    // shared mutex so /metrics readers see the most recent state. Called
    // after every `release` (FIFO completion) per dossier §4 iter-2a
    // step 5. Acquiring the lock costs ~tens of nanoseconds when
    // uncontended; the worker is the sole writer + readers only contend
    // briefly during a Prometheus scrape.
    //
    // Iter-C2c (C2c): generalised to take `&WorkerScheduler` so the
    // InflightBatchedScheduler stats also flow through to /metrics
    // identically.
    let publish_stats = |sched: &WorkerScheduler, snap: &Arc<Mutex<SchedulerStats>>| {
        if let Ok(mut guard) = snap.lock() {
            *guard = sched.stats();
        }
    };

    while let Some(req) = rx.blocking_recv() {
        match req {
            Request::Warmup { reply } => {
                let result = match &mut loaded {
                    LoadedModel::Gemma(g) => warmup_once(g),
                    // Iter-215 MVP: Qwen35 warmup is a no-op (the
                    // chat-completion arm returns 501 immediately so
                    // pre-warming kernels would be wasted work).  /readyz
                    // depends on this returning Ok so the operator-facing
                    // contract — "model is loaded; chat is 501" — surfaces
                    // cleanly rather than a startup failure.
                    LoadedModel::Qwen35(_) => Ok(()),
                    // iter-228a Qwen3-VL text MVP: same shape — chat arm
                    // returns 501; warmup is a no-op so /readyz surfaces
                    // "model is loaded".
                    LoadedModel::Qwen3VlText(_) => Ok(()),
                    LoadedModel::Deepseek4(_) => Ok(()),
                };
                let _ = reply.send(result);
            }
            Request::Generate {
                prompt_tokens,
                params,
                reply,
            } => {
                // ADR-040 Phase C iter-2a (C2b) — Shape A admit→drive→
                // release wrap (dossier §4 iter-2a step 4). Under
                // SerialFifo `admit` is infallible on a fresh adapter
                // (queue_capacity ≥ 1, no in-flight slot) so the
                // QueueFull arm is defensive only — the mpsc channel's
                // backpressure already 429-rejects upstream before
                // reaching worker_run (the 11 handler `tx.try_send`
                // sites at engine.rs:~2832+ map TrySendError::Full to
                // anyhow_bail("queue_full") → HTTP 429). The wrap
                // preserves byte-equivalence (H1/H2 falsifiers) because
                // the inner `generate_*_once` call is unchanged; only
                // pre/post bookkeeping calls were added.
                // ADR-040 §3.5 iter-A5b — compute the real per-request
                // KV byte cost using the spawn-time-cached
                // `kv_bytes_per_token` value (derived from LoadInfo per
                // ADR-040 §3.5 iter-A5b). `0` means "do not enforce"
                // (synthetic test fixtures / arch facts missing) and
                // preserves pre-A5 byte-equivalence verbatim. Production
                // values: (prompt_tokens + max_tokens) ×
                // kv_bytes_per_token, saturating.
                let needed_bytes_admit: u64 =
                    if kv_bytes_per_token == 0 || per_slot_kv_budget_bytes == 0 {
                        0
                    } else {
                        u64::from(prompt_tokens.len() as u32)
                            .saturating_add(u64::from(params.max_tokens as u32))
                            .saturating_mul(kv_bytes_per_token)
                    };
                let admit_req = AdmitRequest {
                    prompt_tokens: prompt_tokens.len() as u32,
                    max_tokens: params.max_tokens as u32,
                    kv_bytes_needed: needed_bytes_admit,
                };
                let admitted = match scheduler.admit(admit_req) {
                    Ok(slot) => slot,
                    Err(AdmitError::QueueFull { .. }) => {
                        let _ = reply.send(Err(anyhow::anyhow!(
                            "ADR-040 C2b: scheduler admit returned QueueFull \
                             in worker_run FifoSerial path (mpsc channel \
                             should have backpressured upstream). \
                             Programming bug — re-check Engine::generate \
                             callsite + handler `tx.try_send` route."
                        )));
                        continue;
                    }
                    // ADR-040 §3.5 iter-A5b — SlotBudgetExceeded surfaces
                    // a `slot_budget_exceeded`-prefixed anyhow error so
                    // the handler layer string-matches parallel to
                    // `queue_full` and emits HTTP 429 + Retry-After per
                    // Decision #19 via ApiError::slot_budget_exceeded.
                    // Iter-A5b wires real per-request kv_bytes_needed at
                    // this admit site so the production path actually
                    // exercises this arm under operator pressure (was
                    // dead code under iter-A5's `kv_bytes_needed: 0`).
                    Err(AdmitError::SlotBudgetExceeded {
                        needed_bytes,
                        budget_bytes,
                    }) => {
                        let _ = reply.send(Err(anyhow::anyhow!(
                            "slot_budget_exceeded: ADR-040 §3.5 A5b — per-slot \
                             KV budget exceeded (needed_bytes={}, budget_bytes={}). \
                             Reduce max_tokens or use a shorter prompt.",
                            needed_bytes,
                            budget_bytes
                        )));
                        continue;
                    }
                    Err(e) => {
                        let _ = reply.send(Err(anyhow::anyhow!(
                            "ADR-040 C2b: scheduler admit failed: {:?}",
                            e
                        )));
                        continue;
                    }
                };

                // ADR-040 Phase C iter-2c (C2c) — Gemma 4 SlotAware
                // typed deferral. Under SlotAware the InflightBatched
                // scheduler may hand out `SlotId(N>0)`. Kernel-level
                // slot routing through `forward_prefill.rs` is
                // iter-C2c-cont scope (gated on Phase B4c per ADR-040
                // §6 + §6.1.21). For now, slot > 0 surfaces a typed
                // `capability_unsupported`-prefixed anyhow error the
                // handler layer maps to HTTP 501 via
                // `ApiError::capability_unsupported`. SerialFifo +
                // SlotId(0) hit the existing forward path unchanged
                // (H23 + H21 byte-equivalence pins).
                if let Some(handle) = admitted.handle {
                    // ADR-040 iter-B4c-kernel iter-1 (2026-05-30) —
                    // Gemma 4 worker hot path LIFT for the Generate
                    // arm onto the persistent multi-seq per-layer
                    // `MultiSeqHbKvBuffers` scaffold (provisioned by
                    // C2c §6.1.21 at spawn time).  C2c added the
                    // dispatch-fork clamp; B4c §6.1.25 refined the
                    // typed-error label; this iter REPLACES the
                    // Generate-arm clamp with the actual scaffold
                    // lift via
                    // `generate_gemma4_once_slot_aware(g,
                    //   &prompt_tokens, &params, registration,
                    //   &mut multi_seq_kv, slot_id)`.
                    //
                    // The take-and-restore borrow pattern at this site
                    // resolves the partial-borrow conflict between
                    // `&mut g.multi_seq_kv` and the dense `&mut
                    // g.lcp_registry` / `&mut g.prompt_cache` accesses
                    // (worker is serial — no concurrent access).
                    //
                    // The other 3 worker arms (GenerateStream / Embed /
                    // GenerateWithSoftTokens) still carry the C2c-cont
                    // typed clamp with relabeled
                    // `iter-B4c-kernel-iter-{3,4,5}` deferral cites —
                    // see §6.1.31 for the iter-1 → iter-{3,4,5}
                    // sequencing decision (iter-2 is the kernel-forward
                    // step itself, typed-deferred inside the
                    // orchestrator).
                    //
                    // SerialFifo + SlotId(0): unchanged (H77 byte-
                    // equivalence pin) — the `slot_id != SlotId(0)`
                    // predicate short-circuits below the lift block so
                    // the existing `generate_once` dispatch at the
                    // `match &mut loaded` below fires verbatim.
                    // SlotAware + SlotId(0): also unchanged (same
                    // predicate; H44 pin preserved).
                    //
                    // Defense-in-depth: if `multi_seq_kv.is_none()` at
                    // SlotId(N>0) (impossible at runtime per the C2c
                    // spawn-arm invariant, but pinned by H81), the
                    // request returns a typed `anyhow::Error` with
                    // operator-grep'able label
                    // `"iter-B4c-kernel iter-1 — multi_seq_kv absent"`.
                    if matches!(loaded, LoadedModel::Gemma(_)) && handle.slot_id != SlotId(0) {
                        let slot_id = handle.slot_id;
                        let LoadedModel::Gemma(g) = &mut loaded else {
                            // Statically unreachable — the matches!
                            // above confirmed Gemma — but Rust's borrow
                            // checker needs the explicit Gemma binding.
                            unreachable!(
                                "ADR-040 iter-B4c-kernel iter-1: \
                                 matches!(Gemma) check passed but bind failed"
                            );
                        };
                        // Take the persistent multi-seq KV out so the
                        // callee gets a clean `&mut Vec<MultiSeqHbKvBuffers>`
                        // without partial-borrow conflicts on the
                        // surrounding `&mut g` accesses.
                        let mut multi_seq = match g.multi_seq_kv.take() {
                            Some(buf) => buf,
                            None => {
                                let _ = reply.send(Err(anyhow::anyhow!(
                                    "capability_unsupported: ADR-040 \
                                     iter-B4c-kernel iter-1 — \
                                     multi_seq_kv is None at SlotId({}) \
                                     for Gemma 4 Generate arm. C2c spawn-arm \
                                     invariant violated (provision_multi_\
                                     seq_kv_for_slot_aware was not called at \
                                     EngineMode::SlotAware spawn time). \
                                     Operator: check spawn_with_mode wiring \
                                     in src/serve/api/engine.rs.",
                                    slot_id.0,
                                )));
                                scheduler.release(handle);
                                publish_stats(&scheduler, &scheduler_stats_snapshot);
                                continue;
                            }
                        };
                        // ADR-040 iter-B4c-kernel iter-2B (2026-05-30) —
                        // PARALLEL take on the production-default hybrid
                        // scaffold sibling iter-C2c-cont (§6.1.33)
                        // provisioned.  `Option<Vec<_>>` shape because the
                        // sibling is `None` when HF2Q_HYBRID_KV=0 (opt-
                        // out); take() leaves the field as `None`
                        // regardless and we restore the original below.
                        // Mirrors the HB take/restore borrow pattern at
                        // line 4824 for the second scaffold.
                        let mut multi_seq_hybrid = g.multi_seq_kv_hybrid.take();
                        // ADR-040 iter-B4c-kernel iter-2D / iter-2C
                        // (§6.1.46) — parallel take on the dense F32 +
                        // legacy 4-bit sibling scaffolds.  Provisioned
                        // IFF the respective env-gate was engaged at
                        // SlotAware spawn time (iter-C2c-cont-cont
                        // Phase 3 / Phase 4); `None` otherwise.
                        let mut multi_seq_dense = g.multi_seq_kv_dense.take();
                        let mut multi_seq_mlx = g.multi_seq_kv_mlx.take();
                        let result = generate_gemma4_once_slot_aware(
                            g,
                            &prompt_tokens,
                            &params,
                            registration.as_ref(),
                            &mut multi_seq,
                            multi_seq_hybrid.as_mut(),
                            multi_seq_dense.as_mut(),
                            multi_seq_mlx.as_mut(),
                            slot_id,
                        );
                        // Put the persistent multi-seq KV back regardless
                        // of result — keeps the spawn-time invariant
                        // (`multi_seq_kv.is_some()` for SlotAware
                        // Gemma 4) intact for the next request.
                        g.multi_seq_kv = Some(multi_seq);
                        // ADR-040 iter-2D + iter-2C: parallel restore
                        // on the dense F32 + legacy 4-bit siblings.
                        g.multi_seq_kv_dense = multi_seq_dense;
                        g.multi_seq_kv_mlx = multi_seq_mlx;
                        // ADR-040 iter-B4c-kernel iter-2B: parallel
                        // restore on the hybrid scaffold sibling.  When
                        // HF2Q_HYBRID_KV=1 (default), this restores the
                        // production-default scaffold; when HF2Q_HYBRID_KV=0
                        // (opt-out), `multi_seq_hybrid` is `None` and we
                        // restore the `None` state (no-op for the
                        // operator-visible state).
                        g.multi_seq_kv_hybrid = multi_seq_hybrid;
                        // Standard post-pattern: bookkeep prefill +
                        // per-token decodes + release.  iter-1's
                        // orchestrator returns a typed CapabilityUnsupported
                        // (iter-2 sub-deferral) — completion_tokens is
                        // 0 on that path so the decode-bookkeeping loop
                        // is a no-op.
                        scheduler.advance_after_prefill(handle, prompt_tokens.len() as u32);
                        if let Ok(ref gr) = result {
                            for _ in 0..gr.completion_tokens {
                                scheduler.advance_after_decode(handle);
                            }
                        }
                        scheduler.release(handle);
                        publish_stats(&scheduler, &scheduler_stats_snapshot);
                        let _ = reply.send(result);
                        continue;
                    }
                    // ADR-040 Phase C iter-C2d-cont-kernel iter-1 (2026-05-29) —
                    // Qwen35 worker hot path FULL LIFT for the Generate arm
                    // onto the persistent multi-seq HybridKvCache. C2d
                    // (§6.1.22) provisions the cache at spawn time; C2d-cont
                    // (§6.1.24) added the typed clamp; this iter REPLACES
                    // the clamp with the actual routing: SlotId(N>0) for
                    // Qwen35 routes through
                    // `engine_qwen35::generate_qwen35_once_slot_aware` which
                    // takes `&mut HybridKvCache` + `SlotId` and dispatches
                    // `forward_gpu_last_logits(.., slot_id)` (B4b §6.1.20
                    // signature). The persistent cache is `take()`-d out of
                    // `Qwen35LoadedModel` for the duration of the call so
                    // the partial-borrow conflict with `qwen.lcp_registry`
                    // etc. is resolved cleanly (worker is serial — no
                    // concurrent access).
                    //
                    // The other 3 worker arms (GenerateStream / Embed /
                    // GenerateWithSoftTokens) still carry the C2d-cont
                    // typed clamp with relabeled
                    // `iter-C2d-cont-kernel-iter-{2,3,4}` deferral cites —
                    // see §6.1.27 for the iter-1 → iter-{2,3,4} sequencing
                    // decision.
                    //
                    // SerialFifo + SlotId(0): unchanged (H51 byte-equivalence
                    // pin) — the slot_id != SlotId(0) predicate short-
                    // circuits below the clamp so the existing per-request
                    // alloc path at the `match &mut loaded` below fires
                    // verbatim. SlotAware + SlotId(0): also unchanged (H52
                    // first-slot pin) — same predicate.
                    //
                    // Defense-in-depth: if `persistent_kv_cache.is_none()`
                    // at SlotId(N>0) (impossible at runtime per the C2d
                    // spawn-arm invariant, but pinned by H55), the request
                    // returns typed `anyhow::Error` with operator-grep'able
                    // label "iter-C2d-cont-kernel — persistent cache absent".
                    if matches!(loaded, LoadedModel::Qwen35(_)) && handle.slot_id != SlotId(0) {
                        let slot_id = handle.slot_id;
                        let LoadedModel::Qwen35(q) = &mut loaded else {
                            // Statically unreachable — the matches! above
                            // confirmed Qwen35 — but Rust's borrow checker
                            // needs the explicit Qwen35 binding.
                            unreachable!(
                                "ADR-040 iter-C2d-cont-kernel iter-1: \
                                 matches!(Qwen35) check passed but bind failed"
                            );
                        };
                        // Take the persistent cache out so callee gets a
                        // clean `&mut HybridKvCache` without partial-borrow
                        // conflicts on the surrounding `&mut q` accesses.
                        let mut persistent = match q.persistent_kv_cache.take() {
                            Some(cache) => cache,
                            None => {
                                let _ = reply.send(Err(anyhow::anyhow!(
                                    "capability_unsupported: ADR-040 \
                                     iter-C2d-cont-kernel iter-1 — \
                                     persistent_kv_cache is None at SlotId({}) \
                                     for Qwen35 Generate arm. C2d spawn-arm \
                                     invariant violated (provision_multi_seq_\
                                     kv_for_slot_aware was not called at \
                                     EngineMode::SlotAware spawn time). \
                                     Operator: check spawn_with_mode wiring \
                                     in src/serve/api/engine.rs.",
                                    slot_id.0,
                                )));
                                scheduler.release(handle);
                                publish_stats(&scheduler, &scheduler_stats_snapshot);
                                continue;
                            }
                        };
                        let result = super::engine_qwen35::generate_qwen35_once_slot_aware(
                            q,
                            &prompt_tokens,
                            &params,
                            registration.as_ref(),
                            &mut persistent,
                            slot_id,
                        );
                        // Put the persistent cache back regardless of
                        // result — keeps the spawn-time invariant
                        // (`persistent_kv_cache.is_some()` for SlotAware
                        // Qwen35) intact for the next request.
                        q.persistent_kv_cache = Some(persistent);
                        // Standard post-pattern: bookkeep prefill +
                        // per-token decodes + release.
                        scheduler.advance_after_prefill(handle, prompt_tokens.len() as u32);
                        if let Ok(ref gr) = result {
                            for _ in 0..gr.completion_tokens {
                                scheduler.advance_after_decode(handle);
                            }
                        }
                        scheduler.release(handle);
                        publish_stats(&scheduler, &scheduler_stats_snapshot);
                        let _ = reply.send(result);
                        continue;
                    }
                    // ADR-040 Phase C iter-C2e (2026-05-30) — Qwen3-VL
                    // SlotAware worker-arm typed clamp for the
                    // Generate arm. Direct mirror of C2c §6.1.21
                    // (Gemma 4) + C2d-cont §6.1.24 (Qwen35) clamps for
                    // the Qwen3-VL text-LM family. The C2e spawn-arm
                    // flip at line ~3654 lets the
                    // `InflightBatchedScheduler` hand out
                    // `SlotId(N>0)` for Qwen3-VL; until iter-228a
                    // lands a real Qwen3-VL forward path past the
                    // 501 sentinel, the worker hot path cannot route
                    // SlotId(N>0) through a persistent KV cache —
                    // there is no persistent cache yet (the iter-9b
                    // naive O(N²) re-prefill loop has no shared
                    // state).
                    //
                    // SerialFifo + SlotId(0): unchanged (H222
                    // byte-equivalence pin) — the `slot_id !=
                    // SlotId(0)` predicate short-circuits below the
                    // clamp so the existing
                    // `generate_qwen3vl_text_once` dispatch fires
                    // verbatim. SlotAware + SlotId(0): also
                    // unchanged (same predicate).
                    //
                    // The worker-arm lift onto the persistent multi-
                    // seq cache lands at **iter-C2e-cont** (post
                    // iter-228a) per ADR-040 §6.1.52.
                    if matches!(loaded, LoadedModel::Qwen3VlText(_)) && handle.slot_id != SlotId(0)
                    {
                        let slot_id = handle.slot_id;
                        // `iter-C2e-cont per ADR-040 §6.1.52` (original
                        // C2e spawn-arm forward-pointer; preserved for
                        // operator-grep compat with H220) — UPGRADED to
                        // `iter-C2e-cont per ADR-040 §6.1.55` (the
                        // structural worker hot path lift closure).
                        //
                        // Take/restore the `slot_aware_max_slots` witness
                        // scalar via the helper + delegate to the
                        // iter-228a `qwen3vl_text_forward_pending_err`
                        // 501 sentinel for verbatim propagation.  See
                        // `Qwen3VlTextLoadedModel::handle_qwen3vl_slot_
                        // aware_n_gt_0_sentinel` docstring + ADR-040
                        // §6.1.55 for the lift rationale.  Once
                        // iter-228a lands the persistent KV cache, the
                        // witness flip is the get-then-put discipline
                        // Qwen35 + Gemma 4 worker arms already use;
                        // sentinel propagation preserved verbatim
                        // (H239 + H240).
                        let result: Result<GenerationResult> =
                            if let LoadedModel::Qwen3VlText(v) = &mut loaded {
                                v.handle_qwen3vl_slot_aware_n_gt_0_sentinel(
                                    slot_id,
                                    "qwen3vl-generate-slot-N",
                                )
                            } else {
                                unreachable!(
                                    "ADR-040 §6.1.55 iter-C2e-cont: matches!(loaded, \
                                 LoadedModel::Qwen3VlText(_)) preconditioned above"
                                )
                            };
                        let _ = reply.send(result);
                        scheduler.release(handle);
                        publish_stats(&scheduler, &scheduler_stats_snapshot);
                        continue;
                    }
                }

                // cfa-iter-C2.5 M1: zero-budget admit (`max_tokens == 0`)
                // returns `RequestSlot { handle: None, .. }` per
                // scheduler.rs `classify_admit`. Preserve pre-ADR-040
                // byte-equivalence by still running `generate_once` (the
                // legacy path applied `params.max_tokens.max(1)` so a
                // single forward + single decode token surfaced), but
                // skip the scheduler bookkeeping entirely — no slot was
                // allocated, so `advance_after_*` + `release` would all
                // be no-ops.
                let result = match &mut loaded {
                    LoadedModel::Gemma(g) => {
                        generate_once(g, &prompt_tokens, &params, registration.as_ref())
                    }
                    // Wedge-3 / iter-216 Phase D: real chat completion via
                    // Qwen35Model::forward_gpu_last_logits + forward_gpu_greedy.
                    LoadedModel::Qwen35(q) => super::engine_qwen35::generate_qwen35_once(
                        q,
                        &prompt_tokens,
                        &params,
                        registration.as_ref(),
                    ),
                    // iter-9b: live dense transformer forward via the
                    // naive O(N²) re-prefill loop in
                    // `engine_qwen3vl::generate_qwen3vl_text_once`.
                    // Replaces the iter-228a 501 sentinel.
                    LoadedModel::Qwen3VlText(q) => {
                        super::engine_qwen3vl::generate_qwen3vl_text_once(
                            q,
                            &prompt_tokens,
                            &params,
                            registration.as_ref(),
                        )
                    }
                    LoadedModel::Deepseek4(d) => super::engine_deepseek4::generate_once(
                        d,
                        &prompt_tokens,
                        &params,
                        registration.as_ref(),
                    ),
                };

                if let Some(handle) = admitted.handle {
                    // ADR-040 C2b post-pattern (dossier §2.9 + §4 iter-2a step 4):
                    // bookkeep the synthetic prefill+decode cycle for
                    // SchedulerStats accuracy. `advance_after_decode` auto-
                    // releases when `tokens_produced >= max_tokens`
                    // (scheduler.rs:506-518); we issue a defensive `release`
                    // afterwards to cover the EOS / stop-string termination
                    // path. Stale-handle calls are silent no-ops per the
                    // iter-2.5 C1 generation-counter discipline.
                    scheduler.advance_after_prefill(handle, prompt_tokens.len() as u32);
                    if let Ok(ref gr) = result {
                        for _ in 0..gr.completion_tokens {
                            scheduler.advance_after_decode(handle);
                        }
                    }
                    scheduler.release(handle);
                }
                publish_stats(&scheduler, &scheduler_stats_snapshot);

                let _ = reply.send(result);
            }
            Request::GenerateStream {
                prompt_tokens,
                params,
                events,
                cancellation_counter,
                soft_tokens,
                deepstack,
                positions_flat,
            } => {
                // ADR-040 C2b admit→drive→release wrap (dossier §4
                // iter-2a step 4). The streaming arm differs from
                // Request::Generate in two ways relevant to scheduler
                // bookkeeping: (a) there is no `reply: oneshot`, so a
                // QueueFull admit failure is communicated via an Error
                // event on the events channel; (b) the streaming
                // function does not return the emitted-token count, so
                // the post-pattern is `advance_after_prefill` + `release`
                // (no per-token `advance_after_decode`). The missing
                // per-token advances are bookkeeping-only — the
                // FifoSchedulerAdapter's auto-release on
                // `tokens_produced >= max_tokens` is moot at max_slots=1
                // (the next request's admit clears the slot regardless),
                // and `SchedulerStats` exports completed_total via
                // `release`, not per-token counters. Byte-equivalence
                // holds because the inner streaming call is unchanged.
                // ADR-040 §3.5 iter-A5b — real per-request KV byte cost
                // computed from the spawn-time-cached `kv_bytes_per_token`.
                // Defense-in-depth at the worker layer: the
                // `Engine::try_admit_budget` pre-stream call in the
                // handler (handlers.rs::chat_completions_stream) has
                // already 429'd over-budget requests BEFORE the SSE body
                // is opened. This second-line check at the worker layer
                // is reachable when the pre-stream check was skipped
                // (non-streaming callers wiring through the same
                // Request::GenerateStream variant in future); it's
                // strictly defensive and never emits to an open SSE
                // stream.
                let needed_bytes_admit: u64 =
                    if kv_bytes_per_token == 0 || per_slot_kv_budget_bytes == 0 {
                        0
                    } else {
                        u64::from(prompt_tokens.len() as u32)
                            .saturating_add(u64::from(params.max_tokens as u32))
                            .saturating_mul(kv_bytes_per_token)
                    };
                let admit_req = AdmitRequest {
                    prompt_tokens: prompt_tokens.len() as u32,
                    max_tokens: params.max_tokens as u32,
                    kv_bytes_needed: needed_bytes_admit,
                };
                let admitted = match scheduler.admit(admit_req) {
                    Ok(slot) => slot,
                    Err(AdmitError::QueueFull { .. }) => {
                        let _ = events.blocking_send(super::sse::GenerationEvent::Error(
                            "ADR-040 C2b: scheduler admit returned \
                                 QueueFull for GenerateStream (mpsc \
                                 backpressure should have rejected upstream)."
                                .to_string(),
                        ));
                        continue;
                    }
                    // ADR-040 §3.5 iter-A5b — surfaces a typed-prefix error
                    // event the SSE layer maps to a clean stream
                    // termination. Reachable only if the pre-stream
                    // `Engine::try_admit_budget` check was bypassed — the
                    // handler call sites all run it first now.
                    Err(AdmitError::SlotBudgetExceeded {
                        needed_bytes,
                        budget_bytes,
                    }) => {
                        let _ = events.blocking_send(super::sse::GenerationEvent::Error(format!(
                            "slot_budget_exceeded: ADR-040 §3.5 A5b — \
                                 per-slot KV budget exceeded for GenerateStream \
                                 (needed_bytes={}, budget_bytes={}). Reduce \
                                 max_tokens or use a shorter prompt.",
                            needed_bytes, budget_bytes
                        )));
                        continue;
                    }
                    Err(e) => {
                        let _ = events.blocking_send(super::sse::GenerationEvent::Error(format!(
                            "ADR-040 C2b: scheduler admit failed for \
                                 GenerateStream: {:?}",
                            e
                        )));
                        continue;
                    }
                };
                // ADR-040 Phase C iter-2c (C2c) — Gemma 4 SlotAware
                // typed deferral for the streaming arm. Mirrors the
                // non-streaming `Request::Generate` guard above; slot >
                // 0 surfaces a `capability_unsupported`-prefixed Error
                // event the SSE layer maps to HTTP 501 via
                // `ApiError::capability_unsupported`. SerialFifo +
                // SlotId(0) hits the existing streaming path unchanged
                // (preserves H23 byte-equivalence for the legacy stream
                // body).
                if let Some(handle) = admitted.handle {
                    // ADR-040 Phase B iter-4c (B4c) refinement of the C2c
                    // streaming arm clamp — symmetric with the C2d-cont
                    // GenerateStream label format. See §6.1.25 for the
                    // full path-decision + label-discipline rationale.
                    //
                    // ADR-040 iter-B4c-kernel iter-3 (2026-05-30) —
                    // Gemma 4 worker hot path LIFT for the GenerateStream
                    // arm onto the persistent multi-seq per-layer
                    // `MultiSeqHbKvBuffers` + sibling
                    // `MultiSeqHybridKvBuffers` scaffolds (provisioned by
                    // C2c §6.1.21 + C2c-cont §6.1.33 at spawn time).
                    // C2c added the dispatch-fork clamp; B4c §6.1.25
                    // refined the typed-error label; iter-1 §6.1.31
                    // labeled the GenerateStream sub-deferral as
                    // `iter-B4c-kernel-iter-3`; this iter REPLACES the
                    // GenerateStream-arm clamp with the actual scaffold
                    // lift via
                    // `generate_stream_gemma4_once_slot_aware(g, .., &mut
                    //   multi_seq, multi_seq_hybrid.as_mut(), slot_id)`.
                    //
                    // Direct mirror of Qwen35 iter-C2d-cont-kernel iter-2
                    // §6.1.28 for the GenerateStream surface; mirror of
                    // iter-B4c-kernel iter-1 §6.1.31 Generate-arm lift
                    // shape for the streaming-event-channel result
                    // surface.
                    //
                    // The take-and-restore borrow pattern at this site
                    // resolves the partial-borrow conflict between
                    // `&mut g.multi_seq_kv` + `&mut g.multi_seq_kv_hybrid`
                    // and the dense `&mut g.lcp_registry` / `&mut
                    // g.prompt_cache` accesses (worker is serial — no
                    // concurrent access).  Parallels iter-1+2B Generate-
                    // arm take/restore at engine.rs:4824-4875.
                    //
                    // The other 2 worker arms (Embed /
                    // GenerateWithSoftTokens) still carry the C2c-cont
                    // typed clamp with `iter-B4c-kernel-iter-{4,5}`
                    // deferral cites — see §6.1.35 for the iter-3 →
                    // iter-{4,5} sequencing decision.
                    //
                    // Vision-augmented streaming (soft_tokens any
                    // non-empty) is deferred to iter-B4c-kernel-iter-5:
                    // the slot-aware fn emits a typed
                    // `capability_unsupported:` error event citing
                    // iter-5 when soft_tokens.is_empty() is false.
                    //
                    // SerialFifo + SlotId(0): unchanged (H104 byte-
                    // equivalence pin) — the `slot_id != SlotId(0)`
                    // predicate short-circuits below the lift block so
                    // the existing `generate_stream_once` dispatch at
                    // the `match &mut loaded` below fires verbatim.
                    // SlotAware + SlotId(0): also unchanged (same
                    // predicate).
                    //
                    // Defense-in-depth: if `multi_seq_kv.is_none()` at
                    // SlotId(N>0) (impossible at runtime per the C2c
                    // spawn-arm invariant), the request emits a typed
                    // Error event with operator-grep'able label
                    // `"iter-B4c-kernel iter-3 — multi_seq_kv absent"`.
                    if matches!(loaded, LoadedModel::Gemma(_)) && handle.slot_id != SlotId(0) {
                        let slot_id = handle.slot_id;
                        let LoadedModel::Gemma(g) = &mut loaded else {
                            unreachable!(
                                "ADR-040 iter-B4c-kernel iter-3: \
                                 matches!(Gemma) check passed but bind failed"
                            );
                        };
                        // Take the persistent multi-seq KV out so the
                        // callee gets a clean `&mut Vec<MultiSeqHbKvBuffers>`
                        // without partial-borrow conflicts on the
                        // surrounding `&mut g` accesses.
                        let mut multi_seq = match g.multi_seq_kv.take() {
                            Some(buf) => buf,
                            None => {
                                let _ = events.blocking_send(super::sse::GenerationEvent::Error(
                                    format!(
                                        "capability_unsupported: ADR-040 \
                                         iter-B4c-kernel iter-3 — \
                                         multi_seq_kv is None at SlotId({}) \
                                         for Gemma 4 GenerateStream arm. C2c \
                                         spawn-arm invariant violated \
                                         (provision_multi_seq_kv_for_slot_aware \
                                         was not called at EngineMode::SlotAware \
                                         spawn time). Operator: check \
                                         spawn_with_mode wiring in \
                                         src/serve/api/engine.rs.",
                                        slot_id.0,
                                    ),
                                ));
                                scheduler.release(handle);
                                publish_stats(&scheduler, &scheduler_stats_snapshot);
                                continue;
                            }
                        };
                        // ADR-040 iter-B4c-kernel iter-3 — PARALLEL take
                        // on the production-default hybrid scaffold
                        // sibling iter-C2c-cont (§6.1.33) provisioned.
                        // `Option<Vec<_>>` shape because the sibling is
                        // `None` when HF2Q_HYBRID_KV=0 (opt-out); take()
                        // leaves the field as `None` regardless and we
                        // restore the original below.  Mirrors the
                        // Generate-arm take/restore pattern at line 4853.
                        let mut multi_seq_hybrid = g.multi_seq_kv_hybrid.take();
                        // ADR-040 iter-2D + iter-2C (§6.1.46) — parallel
                        // take on the dense F32 + legacy 4-bit siblings.
                        let mut multi_seq_dense = g.multi_seq_kv_dense.take();
                        let mut multi_seq_mlx = g.multi_seq_kv_mlx.take();
                        // Build borrowed `SoftTokenInjection<'_>` slices
                        // from the owned `SoftTokenData` (same shape as
                        // the legacy `generate_stream_once` injection
                        // build at line 5337).  The slot-aware fn
                        // surfaces typed error if any extension is
                        // present (vision streaming is iter-5 scope).
                        let injections_slot: Vec<SoftTokenInjection<'_>> = soft_tokens
                            .iter()
                            .map(|d| SoftTokenInjection {
                                range: d.range.clone(),
                                embeddings: &d.embeddings,
                            })
                            .collect();
                        generate_stream_gemma4_once_slot_aware(
                            g,
                            &prompt_tokens,
                            &injections_slot,
                            &params,
                            &events,
                            registration.as_ref(),
                            cancellation_counter.as_deref(),
                            &mut multi_seq,
                            multi_seq_hybrid.as_mut(),
                            multi_seq_dense.as_mut(),
                            multi_seq_mlx.as_mut(),
                            slot_id,
                        );
                        // Put the persistent multi-seq KV back regardless
                        // of outcome — keeps the spawn-time invariant
                        // (`multi_seq_kv.is_some()` for SlotAware Gemma 4)
                        // intact for the next request.
                        g.multi_seq_kv = Some(multi_seq);
                        // ADR-040 iter-B4c-kernel iter-3 — parallel
                        // restore on the hybrid scaffold sibling.  When
                        // HF2Q_HYBRID_KV=1 (default), this restores the
                        // production-default scaffold; when
                        // HF2Q_HYBRID_KV=0 (opt-out), `multi_seq_hybrid`
                        // is `None` and we restore the `None` state.
                        g.multi_seq_kv_hybrid = multi_seq_hybrid;
                        // ADR-040 iter-2D + iter-2C: parallel restore.
                        g.multi_seq_kv_dense = multi_seq_dense;
                        g.multi_seq_kv_mlx = multi_seq_mlx;
                        // Standard post-pattern: bookkeep prefill +
                        // release.  Per-token advance_after_decode is
                        // skipped because the streaming path does not
                        // return the emitted-token count to the worker;
                        // mirrors the existing GenerateStream
                        // post-pattern.  iter-3 today surfaces typed
                        // CapabilityUnsupported on the iter-2-decode
                        // sub-deferral so completion_tokens is implicitly
                        // 0 on that path.
                        scheduler.advance_after_prefill(handle, prompt_tokens.len() as u32);
                        scheduler.release(handle);
                        publish_stats(&scheduler, &scheduler_stats_snapshot);
                        continue;
                    }
                    // ADR-040 Phase C iter-C2d-cont-kernel iter-2 (2026-05-30) —
                    // Qwen35 worker hot path GenerateStream-arm lift onto
                    // the persistent multi-seq `HybridKvCache`. Direct
                    // mirror of iter-1 (§6.1.27 Generate arm) for the
                    // streaming surface. C2d (§6.1.22) provisions the
                    // cache at spawn time; C2d-cont (§6.1.24) added the
                    // typed clamp; iter-1 (§6.1.27) lifted the Generate
                    // arm; this `iter-C2d-cont-kernel-iter-2 per ADR-040
                    // §6.1.28` REPLACES the GenerateStream clamp with the
                    // actual lift via
                    // `engine_qwen35::generate_stream_qwen35_once_extended_slot_aware`
                    // which takes `&mut HybridKvCache` + `SlotId` and
                    // dispatches `forward_gpu_last_logits(.., slot_id)`
                    // (B4b §6.1.20 signature). The persistent cache is
                    // `take()`-d out of `Qwen35LoadedModel` for the
                    // duration of the call so the partial-borrow conflict
                    // with `qwen.lcp_registry` etc. is resolved cleanly
                    // (worker is serial — no concurrent access).
                    //
                    // The other 2 worker arms (Embed /
                    // GenerateWithSoftTokens) still carry the C2d-cont
                    // typed clamp with relabeled
                    // `iter-C2d-cont-kernel-iter-{3,4}` deferral cites —
                    // see §6.1.28 for the iter-2 → iter-{3,4,LCP,G}
                    // sequencing decision.
                    //
                    // Vision-augmented streaming (soft_tokens / deepstack
                    // / positions_flat any non-empty) is deferred to
                    // iter-C2d-cont-kernel-iter-4: the slot-aware fn
                    // emits a typed `capability_unsupported:` error
                    // event citing iter-4 when has_extension is true.
                    //
                    // SerialFifo + SlotId(0): unchanged (H58 byte-equivalence
                    // pin) — the slot_id != SlotId(0) predicate short-
                    // circuits below the clamp so the existing
                    // `generate_stream_qwen35_once_extended` dispatch
                    // fires verbatim. SlotAware + SlotId(0): also
                    // unchanged (same predicate).
                    //
                    // Defense-in-depth: if `persistent_kv_cache.is_none()`
                    // at SlotId(N>0) (impossible at runtime per the C2d
                    // spawn-arm invariant), the request emits a typed
                    // Error event with operator-grep'able label
                    // "iter-C2d-cont-kernel iter-2 — persistent cache absent".
                    if matches!(loaded, LoadedModel::Qwen35(_)) && handle.slot_id != SlotId(0) {
                        let slot_id = handle.slot_id;
                        let LoadedModel::Qwen35(q) = &mut loaded else {
                            unreachable!(
                                "ADR-040 iter-C2d-cont-kernel iter-2: \
                                 matches!(Qwen35) check passed but bind failed"
                            );
                        };
                        // Take the persistent cache out so callee gets a
                        // clean `&mut HybridKvCache` without partial-borrow
                        // conflicts on the surrounding `&mut q` accesses.
                        let mut persistent = match q.persistent_kv_cache.take() {
                            Some(cache) => cache,
                            None => {
                                let _ = events.blocking_send(super::sse::GenerationEvent::Error(
                                    format!(
                                        "capability_unsupported: ADR-040 \
                                         iter-C2d-cont-kernel iter-2 — \
                                         persistent_kv_cache is None at SlotId({}) \
                                         for Qwen35 GenerateStream arm. C2d \
                                         spawn-arm invariant violated \
                                         (provision_multi_seq_kv_for_slot_aware \
                                         was not called at EngineMode::SlotAware \
                                         spawn time). Operator: check \
                                         spawn_with_mode wiring in \
                                         src/serve/api/engine.rs.",
                                        slot_id.0,
                                    ),
                                ));
                                scheduler.release(handle);
                                publish_stats(&scheduler, &scheduler_stats_snapshot);
                                continue;
                            }
                        };
                        // Build borrowed injections for the slot-aware
                        // fn signature mirror; the slot-aware fn
                        // surfaces typed error if any extension is
                        // present (vision streaming is iter-4 scope).
                        let injections_slot: Vec<SoftTokenInjection<'_>> = soft_tokens
                            .iter()
                            .map(|d| SoftTokenInjection {
                                range: d.range.clone(),
                                embeddings: &d.embeddings,
                            })
                            .collect();
                        let ds_borrow_slot: Option<
                            crate::serve::forward_prefill::DeepstackInjection<'_>,
                        > = deepstack.as_ref().map(|d| {
                            crate::serve::forward_prefill::DeepstackInjection {
                                image_token_positions: d.image_token_positions.clone(),
                                chunks: d.chunks.iter().collect(),
                            }
                        });
                        super::engine_qwen35::generate_stream_qwen35_once_extended_slot_aware(
                            q,
                            &prompt_tokens,
                            &injections_slot,
                            ds_borrow_slot.as_ref(),
                            positions_flat.as_deref(),
                            &params,
                            &events,
                            registration.as_ref(),
                            cancellation_counter.as_deref(),
                            &mut persistent,
                            slot_id,
                        );
                        // Put the persistent cache back regardless of
                        // outcome — keeps the spawn-time invariant
                        // (`persistent_kv_cache.is_some()` for SlotAware
                        // Qwen35) intact for the next request.
                        q.persistent_kv_cache = Some(persistent);
                        // Standard post-pattern: bookkeep prefill +
                        // release. Per-token advance_after_decode is
                        // skipped because the streaming path does not
                        // return the emitted-token count to the worker;
                        // mirrors the existing GenerateStream
                        // post-pattern at the bottom of this arm.
                        scheduler.advance_after_prefill(handle, prompt_tokens.len() as u32);
                        scheduler.release(handle);
                        publish_stats(&scheduler, &scheduler_stats_snapshot);
                        continue;
                    }
                    // ADR-040 Phase C iter-C2e (2026-05-30) — Qwen3-VL
                    // SlotAware GenerateStream-arm typed clamp. Direct
                    // mirror of the Generate-arm clamp above + C2c
                    // §6.1.21 GenerateStream + C2d-cont §6.1.24
                    // GenerateStream shape. Surfaces a typed
                    // `capability_unsupported:`-prefixed Error event
                    // onto the SSE channel (the handler maps to a
                    // clean stream termination with a 501-style error
                    // body). See Generate-arm clamp for the full
                    // rationale + iter-228a upstream blocker cite.
                    if matches!(loaded, LoadedModel::Qwen3VlText(_)) && handle.slot_id != SlotId(0)
                    {
                        let slot_id = handle.slot_id;
                        // `iter-C2e-cont per ADR-040 §6.1.52` (preserved
                        // for H220 operator-grep compat) — UPGRADED to
                        // `iter-C2e-cont per ADR-040 §6.1.55` structural
                        // worker hot path lift via the sentinel-aware
                        // helper.  See Generate arm for the full
                        // rationale.
                        let err_msg: String = if let LoadedModel::Qwen3VlText(v) = &mut loaded {
                            // Sentinel arm: the helper returns Err(...)
                            // with the capability_unsupported label.
                            // Extract the Display string for the SSE
                            // Error event (streaming arms route errors
                            // through the SSE channel, not anyhow).
                            let res: Result<()> = v.handle_qwen3vl_slot_aware_n_gt_0_sentinel(
                                slot_id,
                                "qwen3vl-generate-stream-slot-N",
                            );
                            match res {
                                Err(e) => e.to_string(),
                                Ok(()) => unreachable!(
                                    "ADR-040 §6.1.55 iter-C2e-cont: handler \
                                     MUST surface the iter-228a sentinel as Err"
                                ),
                            }
                        } else {
                            unreachable!(
                                "ADR-040 §6.1.55 iter-C2e-cont: matches!(loaded, \
                                 LoadedModel::Qwen3VlText(_)) preconditioned above"
                            )
                        };
                        let _ = events.blocking_send(super::sse::GenerationEvent::Error(err_msg));
                        scheduler.release(handle);
                        publish_stats(&scheduler, &scheduler_stats_snapshot);
                        continue;
                    }
                }

                // cfa-iter-C2.5 M1: zero-budget admit (`max_tokens == 0`)
                // returns `handle: None` (scheduler short-circuits the
                // slot allocation). The streaming arm still drives the
                // legacy `generate_stream_once` body to preserve byte-
                // equivalence — the legacy path emits a Done event with
                // zero deltas — but skips the scheduler bookkeeping.
                let admitted_handle: Option<SlotHandle> = admitted.handle;

                // The streaming path sends every event (Delta / Done / Error)
                // via `events`. Errors stay inside the function — the
                // terminal event is always one of Done/Error, unless the
                // receiver was dropped (client disconnect → early exit).
                // When the early-exit path fires, we bump the cancellation
                // counter if supplied (→ hf2q_sse_cancellations in /metrics).
                //
                // Phase 2c iter-211 W79: build borrowed `SoftTokenInjection<'_>`
                // slices from the owned `SoftTokenData` (channel-friendly Send)
                // mirroring the pattern used by `Request::GenerateWithSoftTokens`
                // above. Empty `soft_tokens` ⇒ identity over the text-only
                // prefill path (the prefill function is already a thin wrapper
                // around `forward_prefill_with_soft_tokens` with an empty
                // slice — see `src/serve/forward_prefill.rs:111-118`).
                //
                // **Wedge-4e (iter-224 row 5)**: extended with borrowed
                // `DeepstackInjection<'_>` constructed from the owned
                // `DeepstackData` (mirrors the non-streaming
                // `Request::GenerateWithSoftTokens` arm). When both
                // `deepstack` and `positions_flat` are `None`, behaviour is
                // byte-identical to the pre-Wedge-4e text-only / pure
                // soft-token streaming path. The Phase-2c soft_token guard
                // that previously sat here has been REMOVED — the Qwen35
                // streaming arm now threads soft_tokens + deepstack +
                // positions through `generate_stream_qwen35_once_extended`.
                let injections: Vec<SoftTokenInjection<'_>> = soft_tokens
                    .iter()
                    .map(|d| SoftTokenInjection {
                        range: d.range.clone(),
                        embeddings: &d.embeddings,
                    })
                    .collect();
                let ds_borrow_stream: Option<
                    crate::serve::forward_prefill::DeepstackInjection<'_>,
                > = deepstack
                    .as_ref()
                    .map(|d| crate::serve::forward_prefill::DeepstackInjection {
                        image_token_positions: d.image_token_positions.clone(),
                        chunks: d.chunks.iter().collect(),
                    });
                match &mut loaded {
                    LoadedModel::Gemma(g) => {
                        // Gemma streaming does not consume `deepstack` or
                        // 3D `positions_flat` (those are Qwen3-VL-only);
                        // the existing entry point ignores both.
                        let _ = (&ds_borrow_stream, &positions_flat);
                        generate_stream_once(
                            g,
                            &prompt_tokens,
                            &injections,
                            &params,
                            &events,
                            registration.as_ref(),
                            cancellation_counter.as_deref(),
                        );
                    }
                    // Wedge-3 / iter-216 Phase D: real streaming chat
                    // completion via Qwen35Model + per-token splitter
                    // routing.  Mirrors the Gemma stream arm shape; tool
                    // calls flow through the close-buffered emitter
                    // (W-B3 incremental shape is a Wedge-4 follow-up).
                    //
                    // **Wedge-4e (iter-224 row 5)**: routes through the
                    // soft-token + deepstack + 3D-positions extended
                    // entry point. When all extensions are empty/None,
                    // behaviour is byte-identical to the pre-Wedge-4e
                    // text-only streaming path (the splitter chain is
                    // MODE-INVARIANT — it operates on token deltas
                    // regardless of prefill source).
                    LoadedModel::Qwen35(q) => {
                        super::engine_qwen35::generate_stream_qwen35_once_extended(
                            q,
                            &prompt_tokens,
                            &injections,
                            ds_borrow_stream.as_ref(),
                            positions_flat.as_deref(),
                            &params,
                            &events,
                            registration.as_ref(),
                            cancellation_counter.as_deref(),
                        );
                    }
                    // iter-228a Qwen3-VL text MVP: emit a single Error
                    // event onto the stream channel carrying the
                    // forward-pending sentinel, so the SSE handler maps
                    // it to a clean stream termination with a 501-style
                    // error body. iter-228b wires the live streaming
                    // forward.
                    LoadedModel::Qwen3VlText(_) => {
                        let _ = (&ds_borrow_stream, &positions_flat);
                        let pending: Result<()> =
                            crate::inference::models::qwen3vl_text::forward::qwen3vl_text_forward_pending_err();
                        if let Err(e) = pending {
                            let _ = events.blocking_send(super::sse::GenerationEvent::Error(
                                format!("{e:#}"),
                            ));
                        }
                    }
                    LoadedModel::Deepseek4(d) => {
                        if !injections.is_empty()
                            || ds_borrow_stream.is_some()
                            || positions_flat.is_some()
                        {
                            let _ = events.blocking_send(super::sse::GenerationEvent::Error(
                                "DeepSeek-V4 does not support multimodal soft-token or \
                                     DeepStack injections"
                                    .to_string(),
                            ));
                        } else {
                            super::engine_deepseek4::generate_stream(
                                d,
                                &prompt_tokens,
                                &params,
                                &events,
                                registration.as_ref(),
                                cancellation_counter.as_deref(),
                            );
                        }
                    }
                }

                // ADR-040 C2b post-pattern — issue the prefill advance
                // (the streaming function ran the prefill internally) +
                // release. Per-token `advance_after_decode` is skipped
                // because the streaming path does not return the emitted-
                // token count to the worker; see the rationale comment at
                // the admit site above. Stale-handle calls on the
                // released handle are silent no-ops per iter-2.5 C1.
                //
                // cfa-iter-C2.5 M1: skip scheduler bookkeeping entirely
                // when the admit was zero-budget (`handle.is_none()`);
                // the scheduler already counted the request as
                // completed-at-admit.
                if let Some(handle) = admitted_handle {
                    scheduler.advance_after_prefill(handle, prompt_tokens.len() as u32);
                    scheduler.release(handle);
                }
                publish_stats(&scheduler, &scheduler_stats_snapshot);
            }
            Request::Embed {
                prompt_tokens,
                reply,
            } => {
                // ADR-040 C2b admit→release wrap (dossier §4 iter-2a
                // step 4). Embed is a single prefill-only forward (no
                // decode loop, no completion tokens). `max_tokens = 0`
                // reflects the embed contract (no sampling budget).
                //
                // cfa-iter-C2.5 M1: under the new admit short-circuit
                // a `max_tokens == 0` admit returns
                // `RequestSlot { handle: None, .. }` and counts as
                // completed-at-admit in `SchedulerStats`. The Embed arm
                // is the canonical zero-budget caller; the prefill
                // forward still runs to produce the embedding result,
                // but no `advance_after_prefill` / `release` is needed
                // (the slot was never allocated).
                // ADR-040 §3.5 iter-A5b — Embed prefill-only KV cost
                // = `prompt_tokens × kv_bytes_per_token` (no decode
                // budget). The per-slot budget check rejects oversized
                // prompts at admit. `kv_bytes_per_token == 0` or
                // `per_slot_kv_budget_bytes == 0` opts out (synthetic
                // fixtures + operators who didn't set
                // `--kv-cache-budget-bytes`).
                let needed_bytes_admit: u64 =
                    if kv_bytes_per_token == 0 || per_slot_kv_budget_bytes == 0 {
                        0
                    } else {
                        u64::from(prompt_tokens.len() as u32).saturating_mul(kv_bytes_per_token)
                    };
                let admit_req = AdmitRequest {
                    prompt_tokens: prompt_tokens.len() as u32,
                    max_tokens: 0,
                    kv_bytes_needed: needed_bytes_admit,
                };
                let admitted = match scheduler.admit(admit_req) {
                    Ok(slot) => slot,
                    Err(AdmitError::QueueFull { .. }) => {
                        let _ = reply.send(Err(anyhow::anyhow!(
                            "ADR-040 C2b: scheduler admit returned QueueFull \
                             for Embed (mpsc backpressure should have rejected \
                             upstream)."
                        )));
                        continue;
                    }
                    // ADR-040 §3.5 iter-A5b — Embed over-budget surfaces
                    // a typed-prefix error that the embeddings handler
                    // string-matches to route to ApiError::slot_budget_exceeded
                    // (HTTP 429 + Retry-After: 1) parallel to queue_full.
                    Err(AdmitError::SlotBudgetExceeded {
                        needed_bytes,
                        budget_bytes,
                    }) => {
                        let _ = reply.send(Err(anyhow::anyhow!(
                            "slot_budget_exceeded: ADR-040 §3.5 A5b — Embed \
                             prompt exceeds per-slot KV budget (needed_bytes={}, \
                             budget_bytes={}). Reduce prompt length.",
                            needed_bytes,
                            budget_bytes
                        )));
                        continue;
                    }
                    Err(e) => {
                        let _ = reply.send(Err(anyhow::anyhow!(
                            "ADR-040 C2b: scheduler admit failed for Embed: {:?}",
                            e
                        )));
                        continue;
                    }
                };

                // ADR-040 Phase B iter-B4c-kernel iter-4 (2026-05-30) —
                // Gemma 4 worker hot path Embed-arm lift onto the
                // persistent multi-seq per-layer `MultiSeqHbKvBuffers`
                // (`g.multi_seq_kv`) + the production-default hybrid
                // F16-K + TQ-HB-V sibling scaffold (`g.multi_seq_kv_hybrid`)
                // instead of the legacy per-request inline `forward_embed_last`
                // on `MlxModelWeights`.  Direct mirror of Qwen35
                // iter-C2d-cont-kernel iter-3 §6.1.29 for the embed
                // surface — same dispatch fork shape (`slot_id != SlotId(0)`
                // predicate at the worker arm), same take-and-restore
                // borrow pattern, same `reset_for_slot` entry+exit
                // discipline.
                //
                // C2c (§6.1.21) added the dispatch-fork clamp; B4c
                // (§6.1.25) refined the typed-error label; iter-1
                // (§6.1.31) labeled the Embed sub-deferral as
                // `iter-B4c-kernel-iter-4`; iter-3 (§6.1.35) lifted the
                // GenerateStream arm; this `iter-B4c-kernel-iter-4 per
                // ADR-040 §6.1.36` REPLACES the Embed clamp with the
                // actual scaffold lift via `embed_gemma4_slot_aware(g,
                // .., &mut multi_seq, multi_seq_hybrid.as_mut(),
                // slot_id)`.
                //
                // The take-and-restore borrow pattern at this site
                // resolves the partial-borrow conflict between
                // `&mut g.multi_seq_kv` + `&mut g.multi_seq_kv_hybrid`
                // and the dense `&mut g.weights` accesses inside
                // `embed_gemma4_slot_aware` (worker is serial — no
                // concurrent access).  Parallels iter-1+2B Generate-
                // arm + iter-3 GenerateStream-arm take/restore at
                // engine.rs:4824-4875 + 5220-5288.
                //
                // The other 1 remaining Gemma 4 worker arm
                // (GenerateWithSoftTokens) still carries the C2c-cont
                // typed clamp with `iter-B4c-kernel-iter-5` deferral
                // cite — see §6.1.36 for the iter-4 → iter-5
                // sequencing decision.
                //
                // SerialFifo + SlotId(0): unchanged (H110 byte-
                // equivalence pin) — the `slot_id != SlotId(0)`
                // predicate short-circuits below the lift block so
                // the existing `g.weights.forward_embed_last(&prompt_tokens,
                // &mut g.ctx)` dispatch at the `match &mut loaded`
                // below fires verbatim.  SlotAware + SlotId(0): also
                // unchanged (same predicate).
                //
                // Defense-in-depth: if `multi_seq_kv.is_none()` at
                // SlotId(N>0) (impossible at runtime per the C2c
                // spawn-arm invariant), the request surfaces a typed
                // `anyhow::Error` with operator-grep'able label
                // `"iter-B4c-kernel iter-4 — multi_seq_kv absent"`.
                if let Some(handle) = admitted.handle {
                    if matches!(loaded, LoadedModel::Gemma(_)) && handle.slot_id != SlotId(0) {
                        let slot_id = handle.slot_id;
                        let LoadedModel::Gemma(g) = &mut loaded else {
                            unreachable!(
                                "ADR-040 iter-B4c-kernel iter-4: \
                                 matches!(Gemma) check passed but bind failed"
                            );
                        };
                        // Take the persistent multi-seq KV out so the
                        // callee gets a clean `&mut Vec<MultiSeqHbKvBuffers>`
                        // without partial-borrow conflicts on the
                        // surrounding `&mut g` accesses.
                        let mut multi_seq = match g.multi_seq_kv.take() {
                            Some(buf) => buf,
                            None => {
                                let _ = reply.send(Err(anyhow::anyhow!(
                                    "capability_unsupported: ADR-040 \
                                     iter-B4c-kernel iter-4 — \
                                     multi_seq_kv is None at SlotId({}) \
                                     for Gemma 4 Embed arm. C2c spawn-arm \
                                     invariant violated (provision_multi_\
                                     seq_kv_for_slot_aware was not called at \
                                     EngineMode::SlotAware spawn time). \
                                     Operator: check spawn_with_mode wiring \
                                     in src/serve/api/engine.rs.",
                                    slot_id.0,
                                )));
                                scheduler.release(handle);
                                publish_stats(&scheduler, &scheduler_stats_snapshot);
                                continue;
                            }
                        };
                        // ADR-040 iter-B4c-kernel iter-4 — PARALLEL take
                        // on the production-default hybrid scaffold
                        // sibling iter-C2c-cont (§6.1.33) provisioned.
                        // `Option<Vec<_>>` shape because the sibling is
                        // `None` when HF2Q_HYBRID_KV=0 (opt-out); take()
                        // leaves the field as `None` regardless and we
                        // restore the original below.  Mirrors the
                        // Generate-arm take/restore pattern at line 4853
                        // + GenerateStream-arm take/restore at line 5251.
                        let mut multi_seq_hybrid = g.multi_seq_kv_hybrid.take();
                        // ADR-040 iter-2D + iter-2C (§6.1.46) — parallel
                        // take on the dense F32 + legacy 4-bit siblings.
                        let mut multi_seq_dense = g.multi_seq_kv_dense.take();
                        let mut multi_seq_mlx = g.multi_seq_kv_mlx.take();
                        let result = embed_gemma4_slot_aware(
                            g,
                            &prompt_tokens,
                            &mut multi_seq,
                            multi_seq_hybrid.as_mut(),
                            multi_seq_dense.as_mut(),
                            multi_seq_mlx.as_mut(),
                            slot_id,
                        );
                        // Put the persistent multi-seq KV back regardless
                        // of result — keeps the spawn-time invariant
                        // (`multi_seq_kv.is_some()` for SlotAware Gemma 4)
                        // intact for the next request.
                        g.multi_seq_kv = Some(multi_seq);
                        // ADR-040 iter-B4c-kernel iter-4: parallel
                        // restore on the hybrid scaffold sibling.  When
                        // HF2Q_HYBRID_KV=1 (default), this restores the
                        // production-default scaffold; when
                        // HF2Q_HYBRID_KV=0 (opt-out), `multi_seq_hybrid`
                        // is `None` and we restore the `None` state.
                        g.multi_seq_kv_hybrid = multi_seq_hybrid;
                        // ADR-040 iter-2D + iter-2C: parallel restore.
                        g.multi_seq_kv_dense = multi_seq_dense;
                        g.multi_seq_kv_mlx = multi_seq_mlx;
                        // Standard post-pattern: bookkeep prefill +
                        // release.  Embed has no decode loop, so no
                        // per-token `advance_after_decode` calls (mirror
                        // of the legacy Embed post-pattern at the bottom
                        // of this arm + Qwen35 iter-3 §6.1.29 shape).
                        scheduler.advance_after_prefill(handle, prompt_tokens.len() as u32);
                        scheduler.release(handle);
                        publish_stats(&scheduler, &scheduler_stats_snapshot);
                        let _ = reply.send(result);
                        continue;
                    }
                    // ADR-040 Phase C iter-C2d-cont-kernel iter-3 (2026-05-30) —
                    // Qwen35 worker hot path Embed-arm lift onto the
                    // persistent multi-seq `HybridKvCache`. Direct mirror
                    // of iter-1 (§6.1.27 Generate arm) + iter-2 (§6.1.28
                    // GenerateStream arm) for the embed surface. C2d
                    // (§6.1.22) provisions the cache at spawn time;
                    // C2d-cont (§6.1.24) added the typed clamp; iter-1
                    // (§6.1.27) lifted the Generate arm; iter-2 (§6.1.28)
                    // lifted the GenerateStream arm; this `iter-C2d-cont-
                    // kernel-iter-3 per ADR-040 §6.1.29` REPLACES the
                    // Embed clamp with the actual lift via
                    // `engine_qwen35::embed_qwen35_slot_aware` which
                    // takes `&mut HybridKvCache` + `SlotId` and
                    // dispatches `forward_embed_last(.., slot_id)` (B4b
                    // §6.1.20 signature). The persistent cache is
                    // `take()`-d out of `Qwen35LoadedModel` for the
                    // duration of the call so the partial-borrow
                    // conflict with `qwen.lcp_registry` etc. is resolved
                    // cleanly (worker is serial — no concurrent access).
                    //
                    // The other 1 worker arm (GenerateWithSoftTokens)
                    // still carries the C2d-cont typed clamp with
                    // relabeled `iter-C2d-cont-kernel-iter-4` deferral
                    // cite — see §6.1.29 for the iter-3 → iter-4
                    // sequencing decision.
                    //
                    // SerialFifo + SlotId(0): unchanged (H64 byte-
                    // equivalence pin) — the `slot_id != SlotId(0)`
                    // predicate short-circuits below the lift block so
                    // the existing `embed_qwen35` dispatch fires
                    // verbatim. SlotAware + SlotId(0): also unchanged
                    // (same predicate).
                    //
                    // Defense-in-depth: if `persistent_kv_cache.is_none()`
                    // at SlotId(N>0) (impossible at runtime per the C2d
                    // spawn-arm invariant), the request surfaces a typed
                    // `anyhow::Error` with operator-grep'able label
                    // "iter-C2d-cont-kernel iter-3 — persistent cache absent".
                    if matches!(loaded, LoadedModel::Qwen35(_)) && handle.slot_id != SlotId(0) {
                        let slot_id = handle.slot_id;
                        let LoadedModel::Qwen35(q) = &mut loaded else {
                            unreachable!(
                                "ADR-040 iter-C2d-cont-kernel iter-3: \
                                 matches!(Qwen35) check passed but bind failed"
                            );
                        };
                        // Take the persistent cache out so callee gets a
                        // clean `&mut HybridKvCache` without partial-borrow
                        // conflicts on the surrounding `&mut q` accesses.
                        let mut persistent = match q.persistent_kv_cache.take() {
                            Some(cache) => cache,
                            None => {
                                let _ = reply.send(Err(anyhow::anyhow!(
                                    "capability_unsupported: ADR-040 \
                                     iter-C2d-cont-kernel iter-3 — \
                                     persistent_kv_cache is None at SlotId({}) \
                                     for Qwen35 Embed arm. C2d spawn-arm \
                                     invariant violated (provision_multi_seq_\
                                     kv_for_slot_aware was not called at \
                                     EngineMode::SlotAware spawn time). \
                                     Operator: check spawn_with_mode wiring \
                                     in src/serve/api/engine.rs.",
                                    slot_id.0,
                                )));
                                scheduler.release(handle);
                                publish_stats(&scheduler, &scheduler_stats_snapshot);
                                continue;
                            }
                        };
                        let result = super::engine_qwen35::embed_qwen35_slot_aware(
                            q,
                            &prompt_tokens,
                            &mut persistent,
                            slot_id,
                        );
                        // Put the persistent cache back regardless of
                        // result — keeps the spawn-time invariant
                        // (`persistent_kv_cache.is_some()` for SlotAware
                        // Qwen35) intact for the next request.
                        q.persistent_kv_cache = Some(persistent);
                        // Standard post-pattern: bookkeep prefill +
                        // release. Embed has no decode loop, so no
                        // per-token `advance_after_decode` calls (mirror
                        // of the legacy Embed post-pattern at the bottom
                        // of this arm).
                        scheduler.advance_after_prefill(handle, prompt_tokens.len() as u32);
                        scheduler.release(handle);
                        publish_stats(&scheduler, &scheduler_stats_snapshot);
                        let _ = reply.send(result);
                        continue;
                    }
                    // ADR-040 Phase C iter-C2e (2026-05-30) — Qwen3-VL
                    // SlotAware Embed-arm typed clamp. Direct mirror
                    // of the Generate / GenerateStream Qwen3VL clamps
                    // above + C2c §6.1.21 Embed + C2d-cont §6.1.24
                    // Embed shape. See Generate-arm clamp for the
                    // full rationale + iter-228a upstream blocker
                    // cite.
                    if matches!(loaded, LoadedModel::Qwen3VlText(_)) && handle.slot_id != SlotId(0)
                    {
                        let slot_id = handle.slot_id;
                        // `iter-C2e-cont per ADR-040 §6.1.52` (preserved
                        // for H220 operator-grep compat) — UPGRADED to
                        // `iter-C2e-cont per ADR-040 §6.1.55` structural
                        // worker hot path lift via the sentinel-aware
                        // helper.  See Generate arm for the full
                        // rationale.
                        let result: Result<Vec<f32>> =
                            if let LoadedModel::Qwen3VlText(v) = &mut loaded {
                                v.handle_qwen3vl_slot_aware_n_gt_0_sentinel(
                                    slot_id,
                                    "qwen3vl-embed-slot-N",
                                )
                            } else {
                                unreachable!(
                                    "ADR-040 §6.1.55 iter-C2e-cont: matches!(loaded, \
                                 LoadedModel::Qwen3VlText(_)) preconditioned above"
                                )
                            };
                        let _ = reply.send(result);
                        scheduler.release(handle);
                        publish_stats(&scheduler, &scheduler_stats_snapshot);
                        continue;
                    }
                }

                // Single-shot pooled embedding (Last pooling).  The
                // worker holds &mut LoadedModel, so prefill's mutation of
                // self.activations + self.dense_kvs is fine here — it
                // can't race with a concurrent generate call because the
                // worker is serial.
                let result = match &mut loaded {
                    LoadedModel::Gemma(g) => {
                        g.weights.forward_embed_last(&prompt_tokens, &mut g.ctx)
                    }
                    // Wedge-3 / iter-216 Phase D: real chat-as-embedder via
                    // Qwen35Model::forward_embed_last (Phase A).
                    LoadedModel::Qwen35(q) => super::engine_qwen35::embed_qwen35(q, &prompt_tokens),
                    // iter-228a Qwen3-VL text MVP: embed surface joins
                    // chat in returning the forward-pending sentinel.
                    // iter-228b lands a real `forward_embed_last`
                    // (mirroring engine_qwen35::embed_qwen35).
                    LoadedModel::Qwen3VlText(_) => {
                        crate::inference::models::qwen3vl_text::forward::qwen3vl_text_forward_pending_err()
                    }
                    LoadedModel::Deepseek4(_) => Err(anyhow::anyhow!(
                        "embeddings are not supported by the DeepSeek-V4 generative runtime"
                    )),
                };

                if let Some(handle) = admitted.handle {
                    // Defensive: under the iter-C2.5 M1 short-circuit a
                    // `max_tokens == 0` admit always returns `handle: None`
                    // for Embed, but if a future caller flips the contract
                    // to pass non-zero max_tokens this preserves the
                    // existing release pattern.
                    scheduler.advance_after_prefill(handle, prompt_tokens.len() as u32);
                    scheduler.release(handle);
                }
                publish_stats(&scheduler, &scheduler_stats_snapshot);

                let _ = reply.send(result);
            }
            Request::GenerateWithSoftTokens {
                prompt_tokens,
                soft_tokens,
                params,
                deepstack,
                positions_flat,
                reply,
            } => {
                // ADR-040 C2b admit→drive→release wrap (dossier §4
                // iter-2a step 4). Vision-aware generate shares the
                // synthetic prefill+decode bookkeeping shape with
                // Request::Generate (single prompt prefill, then
                // tokens_produced decodes).
                // ADR-040 §3.5 iter-A5b — real per-request KV byte cost
                // for the vision-aware generate path. Shares the same
                // (prompt_tokens + max_tokens) × kv_bytes_per_token
                // formula as the text-only Generate arm. `0` opts out
                // (synthetic fixtures + unset --kv-cache-budget-bytes).
                let needed_bytes_admit: u64 =
                    if kv_bytes_per_token == 0 || per_slot_kv_budget_bytes == 0 {
                        0
                    } else {
                        u64::from(prompt_tokens.len() as u32)
                            .saturating_add(u64::from(params.max_tokens as u32))
                            .saturating_mul(kv_bytes_per_token)
                    };
                let admit_req = AdmitRequest {
                    prompt_tokens: prompt_tokens.len() as u32,
                    max_tokens: params.max_tokens as u32,
                    kv_bytes_needed: needed_bytes_admit,
                };
                let admitted = match scheduler.admit(admit_req) {
                    Ok(slot) => slot,
                    Err(AdmitError::QueueFull { .. }) => {
                        let _ = reply.send(Err(anyhow::anyhow!(
                            "ADR-040 C2b: scheduler admit returned QueueFull \
                             for GenerateWithSoftTokens (mpsc backpressure \
                             should have rejected upstream)."
                        )));
                        continue;
                    }
                    // ADR-040 §3.5 iter-A5b — typed-prefix error for
                    // the handler-side 429 mapping (same shape as the
                    // text-only Generate arm).
                    Err(AdmitError::SlotBudgetExceeded {
                        needed_bytes,
                        budget_bytes,
                    }) => {
                        let _ = reply.send(Err(anyhow::anyhow!(
                            "slot_budget_exceeded: ADR-040 §3.5 A5b — scheduler rejected \
                             GenerateWithSoftTokens — per-slot KV budget \
                             exceeded (needed_bytes={}, budget_bytes={}). \
                             Reduce max_tokens or use a shorter prompt.",
                            needed_bytes,
                            budget_bytes
                        )));
                        continue;
                    }
                    Err(e) => {
                        let _ = reply.send(Err(anyhow::anyhow!(
                            "ADR-040 C2b: scheduler admit failed for \
                             GenerateWithSoftTokens: {:?}",
                            e
                        )));
                        continue;
                    }
                };

                // ADR-040 Phase C iter-2c (C2c) — Gemma 4 SlotAware
                // typed deferral for the vision-aware Generate arm.
                // Mirrors the text-only Generate guard. Vision soft-token
                // overrides at slot > 0 still need kernel slot routing
                // in `forward_prefill.rs` (iter-C2c-cont scope).
                if let Some(handle) = admitted.handle {
                    // ADR-040 Phase B iter-B4c-kernel iter-5 (2026-05-30) —
                    // Gemma 4 worker hot path GenerateWithSoftTokens-arm
                    // lift onto the persistent multi-seq per-layer
                    // `MultiSeqHbKvBuffers` (`g.multi_seq_kv`) + the
                    // production-default hybrid F16-K + TQ-HB-V sibling
                    // scaffold (`g.multi_seq_kv_hybrid`) instead of the
                    // legacy per-request inline `generate_once_with_soft_tokens`
                    // dispatch.  Direct mirror of Qwen35 iter-C2d-cont-
                    // kernel iter-4 §6.1.30 for the Gemma 4 vision-aware
                    // soft-token surface — same dispatch fork shape
                    // (`slot_id != SlotId(0)` predicate at the worker
                    // arm), same take-and-restore borrow pattern on
                    // BOTH scaffolds, same `reset_for_slot` entry+exit
                    // discipline on BOTH scaffolds.
                    //
                    // C2c (§6.1.21) added the dispatch-fork clamp; B4c
                    // (§6.1.25) refined the typed-error label; iter-1
                    // (§6.1.31) labeled the SoftTokens sub-deferral as
                    // `iter-B4c-kernel-iter-5`; iter-3 (§6.1.35) lifted
                    // the GenerateStream arm; iter-4 (§6.1.36) lifted
                    // the Embed arm; this `iter-B4c-kernel-iter-5 per
                    // ADR-040 §6.1.37` REPLACES the SoftTokens clamp
                    // with the actual scaffold lift via
                    // `generate_gemma4_once_with_soft_tokens_slot_aware(g,
                    //   .., &mut multi_seq, multi_seq_hybrid.as_mut(),
                    //   slot_id)`.
                    //
                    // iter-5 is the **TERMINAL Gemma 4 worker-arm lift**
                    // — post-iter-5 ALL FOUR Gemma 4 worker arms route
                    // through the persistent multi-seq scaffolds at
                    // SlotId(N>0).  The Gemma 4 worker-arm lift arc is
                    // COMPLETE.  Surviving sub-deferrals (iter-2A-cont,
                    // iter-2C, iter-2D, iter-2B-xlen, iter-2-decode,
                    // iter-LCP, iter-G) are orthogonal kernel-side
                    // refactors, NOT arm lifts.
                    //
                    // The take-and-restore borrow pattern at this site
                    // resolves the partial-borrow conflict between
                    // `&mut g.multi_seq_kv` + `&mut g.multi_seq_kv_hybrid`
                    // and the dense `&mut g.lcp_registry` / `&mut
                    // g.prompt_cache` accesses (worker is serial — no
                    // concurrent access).  Parallels iter-1+2B Generate
                    // arm take/restore + iter-3 GenerateStream-arm
                    // take/restore + iter-4 Embed-arm take/restore.
                    //
                    // SerialFifo + SlotId(0): unchanged (H116 byte-
                    // equivalence pin) — the `slot_id != SlotId(0)`
                    // predicate short-circuits below the lift block so
                    // the existing `generate_once_with_soft_tokens`
                    // dispatch at the `match &mut loaded` below fires
                    // verbatim.  SlotAware + SlotId(0): also unchanged
                    // (same predicate).
                    //
                    // Defense-in-depth: if `multi_seq_kv.is_none()` at
                    // SlotId(N>0) (impossible at runtime per the C2c
                    // spawn-arm invariant), the request surfaces a typed
                    // `anyhow::Error` with operator-grep'able label
                    // `"iter-B4c-kernel iter-5 — multi_seq_kv absent"`.
                    if matches!(loaded, LoadedModel::Gemma(_)) && handle.slot_id != SlotId(0) {
                        let slot_id = handle.slot_id;
                        let LoadedModel::Gemma(g) = &mut loaded else {
                            unreachable!(
                                "ADR-040 iter-B4c-kernel iter-5: \
                                 matches!(Gemma) check passed but bind failed"
                            );
                        };
                        // Take the persistent multi-seq KV out so the
                        // callee gets a clean `&mut Vec<MultiSeqHbKvBuffers>`
                        // without partial-borrow conflicts on the
                        // surrounding `&mut g` accesses.
                        let mut multi_seq = match g.multi_seq_kv.take() {
                            Some(buf) => buf,
                            None => {
                                let _ = reply.send(Err(anyhow::anyhow!(
                                    "capability_unsupported: ADR-040 \
                                     iter-B4c-kernel iter-5 — \
                                     multi_seq_kv is None at SlotId({}) \
                                     for Gemma 4 GenerateWithSoftTokens \
                                     arm. C2c spawn-arm invariant violated \
                                     (provision_multi_seq_kv_for_slot_aware \
                                     was not called at EngineMode::SlotAware \
                                     spawn time). Operator: check \
                                     spawn_with_mode wiring in \
                                     src/serve/api/engine.rs.",
                                    slot_id.0,
                                )));
                                scheduler.release(handle);
                                publish_stats(&scheduler, &scheduler_stats_snapshot);
                                continue;
                            }
                        };
                        // ADR-040 iter-B4c-kernel iter-5 — PARALLEL take
                        // on the production-default hybrid scaffold
                        // sibling iter-C2c-cont (§6.1.33) provisioned.
                        // `Option<Vec<_>>` shape because the sibling is
                        // `None` when HF2Q_HYBRID_KV=0 (opt-out); take()
                        // leaves the field as `None` regardless and we
                        // restore the original below.  Mirrors the
                        // Generate-arm take/restore pattern + iter-3
                        // GenerateStream-arm + iter-4 Embed-arm patterns.
                        let mut multi_seq_hybrid = g.multi_seq_kv_hybrid.take();
                        // ADR-040 iter-2D + iter-2C (§6.1.46) — parallel
                        // take on the dense F32 + legacy 4-bit siblings.
                        let mut multi_seq_dense = g.multi_seq_kv_dense.take();
                        let mut multi_seq_mlx = g.multi_seq_kv_mlx.take();
                        // Build borrowed `SoftTokenInjection<'_>` slices
                        // from the owned `SoftTokenData` (same shape as
                        // the legacy `generate_once_with_soft_tokens`
                        // injection build below + iter-3 GenerateStream-
                        // arm injection build at engine.rs:5258-5264).
                        let injections_slot: Vec<SoftTokenInjection<'_>> = soft_tokens
                            .iter()
                            .map(|d| SoftTokenInjection {
                                range: d.range.clone(),
                                embeddings: &d.embeddings,
                            })
                            .collect();
                        // Gemma 4 SoftTokens-arm scope: deepstack +
                        // positions_flat are Qwen3-VL specific and
                        // intentionally NOT consumed here (the non-
                        // slot-aware sibling at engine.rs:6144-6155
                        // makes the same choice — Gemma 4 falls back
                        // to the soft-token-only entry).  Threading
                        // deepstack/positions_flat through would
                        // require a deepstack-aware Gemma 4 forward
                        // kernel, which does NOT exist (the Wedge-4d
                        // DeepStack pipeline is Qwen35/Qwen3-VL only).
                        let _ = &deepstack;
                        let _ = &positions_flat;
                        let result = generate_gemma4_once_with_soft_tokens_slot_aware(
                            g,
                            &prompt_tokens,
                            &injections_slot,
                            &params,
                            registration.as_ref(),
                            &mut multi_seq,
                            multi_seq_hybrid.as_mut(),
                            multi_seq_dense.as_mut(),
                            multi_seq_mlx.as_mut(),
                            slot_id,
                        );
                        // Put the persistent multi-seq KV back regardless
                        // of result — keeps the spawn-time invariant
                        // (`multi_seq_kv.is_some()` for SlotAware Gemma 4)
                        // intact for the next request.
                        g.multi_seq_kv = Some(multi_seq);
                        // ADR-040 iter-B4c-kernel iter-5: parallel
                        // restore on the hybrid scaffold sibling.  When
                        // HF2Q_HYBRID_KV=1 (default), this restores the
                        // production-default scaffold; when
                        // HF2Q_HYBRID_KV=0 (opt-out), `multi_seq_hybrid`
                        // is `None` and we restore the `None` state.
                        g.multi_seq_kv_hybrid = multi_seq_hybrid;
                        // ADR-040 iter-2D + iter-2C: parallel restore.
                        g.multi_seq_kv_dense = multi_seq_dense;
                        g.multi_seq_kv_mlx = multi_seq_mlx;
                        // Standard post-pattern: bookkeep prefill +
                        // per-token decodes + release.  iter-5 today
                        // surfaces typed CapabilityUnsupported on the
                        // iter-2-decode sub-deferral so completion_tokens
                        // is implicitly 0 on that path (the decode-
                        // bookkeeping loop is a no-op on Err).
                        scheduler.advance_after_prefill(handle, prompt_tokens.len() as u32);
                        if let Ok(ref gr) = result {
                            for _ in 0..gr.completion_tokens {
                                scheduler.advance_after_decode(handle);
                            }
                        }
                        scheduler.release(handle);
                        publish_stats(&scheduler, &scheduler_stats_snapshot);
                        let _ = reply.send(result);
                        continue;
                    }
                    // ADR-040 Phase C iter-C2d-cont-kernel iter-4 (2026-05-30) —
                    // Qwen35 worker hot path GenerateWithSoftTokens-arm lift
                    // onto the persistent multi-seq `HybridKvCache`. Direct
                    // mirror of iter-1 (§6.1.27 Generate arm) + iter-2
                    // (§6.1.28 GenerateStream arm) + iter-3 (§6.1.29 Embed
                    // arm) for the vision-aware soft-token surface. C2d
                    // (§6.1.22) provisions the cache at spawn time; C2d-cont
                    // (§6.1.24) added the typed clamp; iter-1/2/3 lifted
                    // Generate / GenerateStream / Embed; this
                    // `iter-C2d-cont-kernel-iter-4 per ADR-040 §6.1.30`
                    // REPLACES the GenerateWithSoftTokens clamp with the
                    // actual lift via
                    // `engine_qwen35::generate_qwen35_once_with_soft_tokens_slot_aware`
                    // (soft-tokens-only) or
                    // `engine_qwen35::generate_qwen35_once_with_soft_tokens_and_deepstack_slot_aware`
                    // (deepstack / 3D positions). Both fns take `&mut
                    // HybridKvCache` + `SlotId` and dispatch the
                    // `forward_gpu_last_logits_with_soft_tokens*` family
                    // (B4b §6.1.20 signature). The persistent cache is
                    // `take()`-d out of `Qwen35LoadedModel` for the
                    // duration of the call so the partial-borrow conflict
                    // with `qwen.lcp_registry` etc. is resolved cleanly
                    // (worker is serial — no concurrent access).
                    //
                    // iter-4 is the TERMINAL Qwen35 worker-arm lift —
                    // post-iter-4 ALL FOUR Qwen35 worker arms (Generate +
                    // GenerateStream + Embed + GenerateWithSoftTokens)
                    // route through the persistent multi-seq cache at
                    // SlotId(N>0). The remaining sub-deferrals
                    // (iter-LCP for slot-aware LCP / chunked-prefill
                    // codec; iter-G for slot-aware
                    // `forward_gpu_greedy` fast-path) are orthogonal
                    // optimizations, not arm lifts.
                    //
                    // SerialFifo + SlotId(0): unchanged (H70 byte-
                    // equivalence pin) — the `slot_id != SlotId(0)`
                    // predicate short-circuits below the lift block so
                    // the existing per-request alloc path at the
                    // `match &mut loaded` below fires verbatim.
                    // SlotAware + SlotId(0): also unchanged (same
                    // predicate).
                    //
                    // Defense-in-depth: if `persistent_kv_cache.is_none()`
                    // at SlotId(N>0) (impossible at runtime per the C2d
                    // spawn-arm invariant), the request surfaces a typed
                    // `anyhow::Error` with operator-grep'able label
                    // "iter-C2d-cont-kernel iter-4 — persistent cache
                    // absent".
                    if matches!(loaded, LoadedModel::Qwen35(_)) && handle.slot_id != SlotId(0) {
                        let slot_id = handle.slot_id;
                        let LoadedModel::Qwen35(q) = &mut loaded else {
                            unreachable!(
                                "ADR-040 iter-C2d-cont-kernel iter-4: \
                                 matches!(Qwen35) check passed but bind failed"
                            );
                        };
                        // Take the persistent cache out so callee gets a
                        // clean `&mut HybridKvCache` without partial-borrow
                        // conflicts on the surrounding `&mut q` accesses.
                        let mut persistent = match q.persistent_kv_cache.take() {
                            Some(cache) => cache,
                            None => {
                                let _ = reply.send(Err(anyhow::anyhow!(
                                    "capability_unsupported: ADR-040 \
                                     iter-C2d-cont-kernel iter-4 — \
                                     persistent_kv_cache is None at SlotId({}) \
                                     for Qwen35 GenerateWithSoftTokens arm. \
                                     C2d spawn-arm invariant violated \
                                     (provision_multi_seq_kv_for_slot_aware \
                                     was not called at EngineMode::SlotAware \
                                     spawn time). Operator: check \
                                     spawn_with_mode wiring in \
                                     src/serve/api/engine.rs.",
                                    slot_id.0,
                                )));
                                scheduler.release(handle);
                                publish_stats(&scheduler, &scheduler_stats_snapshot);
                                continue;
                            }
                        };
                        // Build borrowed injections for the slot-aware
                        // fn signature mirror (same shape as iter-2's
                        // GenerateStream-arm lift fork).
                        let injections_slot: Vec<SoftTokenInjection<'_>> = soft_tokens
                            .iter()
                            .map(|d| SoftTokenInjection {
                                range: d.range.clone(),
                                embeddings: &d.embeddings,
                            })
                            .collect();
                        let ds_borrow_slot: Option<
                            crate::serve::forward_prefill::DeepstackInjection<'_>,
                        > = deepstack.as_ref().map(|d| {
                            crate::serve::forward_prefill::DeepstackInjection {
                                image_token_positions: d.image_token_positions.clone(),
                                chunks: d.chunks.iter().collect(),
                            }
                        });
                        // Dispatch: deepstack / positions present →
                        // deepstack-aware slot-aware fn; else soft-tokens-
                        // only slot-aware fn. Mirrors the non-slot-aware
                        // arm's dispatch shape at engine.rs:5511 below.
                        let result = if ds_borrow_slot.is_some() || positions_flat.is_some() {
                            super::engine_qwen35::generate_qwen35_once_with_soft_tokens_and_deepstack_slot_aware(
                                q,
                                &prompt_tokens,
                                &injections_slot,
                                ds_borrow_slot.as_ref(),
                                positions_flat.as_deref(),
                                &params,
                                registration.as_ref(),
                                &mut persistent,
                                slot_id,
                            )
                        } else {
                            super::engine_qwen35::generate_qwen35_once_with_soft_tokens_slot_aware(
                                q,
                                &prompt_tokens,
                                &injections_slot,
                                &params,
                                registration.as_ref(),
                                &mut persistent,
                                slot_id,
                            )
                        };
                        // Put the persistent cache back regardless of
                        // result — keeps the spawn-time invariant
                        // (`persistent_kv_cache.is_some()` for SlotAware
                        // Qwen35) intact for the next request.
                        q.persistent_kv_cache = Some(persistent);
                        // Standard post-pattern: bookkeep prefill +
                        // per-token decodes + release. Mirror of iter-1
                        // Generate-arm post-pattern.
                        scheduler.advance_after_prefill(handle, prompt_tokens.len() as u32);
                        if let Ok(ref gr) = result {
                            for _ in 0..gr.completion_tokens {
                                scheduler.advance_after_decode(handle);
                            }
                        }
                        scheduler.release(handle);
                        publish_stats(&scheduler, &scheduler_stats_snapshot);
                        let _ = reply.send(result);
                        continue;
                    }
                    // ADR-040 Phase C iter-C2e (2026-05-30) — Qwen3-VL
                    // SlotAware GenerateWithSoftTokens-arm typed
                    // clamp. Direct mirror of the Generate /
                    // GenerateStream / Embed Qwen3VL clamps above + C2c
                    // §6.1.21 SoftTokens + C2d-cont §6.1.24 SoftTokens
                    // shape. See Generate-arm clamp for the full
                    // rationale + iter-228a upstream blocker cite.
                    if matches!(loaded, LoadedModel::Qwen3VlText(_)) && handle.slot_id != SlotId(0)
                    {
                        let slot_id = handle.slot_id;
                        // `iter-C2e-cont per ADR-040 §6.1.52` (preserved
                        // for H220 operator-grep compat) — UPGRADED to
                        // `iter-C2e-cont per ADR-040 §6.1.55` structural
                        // worker hot path lift via the sentinel-aware
                        // helper.  See Generate arm for the full
                        // rationale.
                        let result: Result<GenerationResult> =
                            if let LoadedModel::Qwen3VlText(v) = &mut loaded {
                                v.handle_qwen3vl_slot_aware_n_gt_0_sentinel(
                                    slot_id,
                                    "qwen3vl-generate-with-soft-tokens-slot-N",
                                )
                            } else {
                                unreachable!(
                                    "ADR-040 §6.1.55 iter-C2e-cont: matches!(loaded, \
                                 LoadedModel::Qwen3VlText(_)) preconditioned above"
                                )
                            };
                        let _ = reply.send(result);
                        scheduler.release(handle);
                        publish_stats(&scheduler, &scheduler_stats_snapshot);
                        continue;
                    }
                }

                // cfa-iter-C2.5 M1: zero-budget admit (`max_tokens == 0`)
                // returns `handle: None`; preserve pre-ADR-040 behaviour
                // by still running the legacy `generate_*_with_soft_tokens*`
                // body (which applies `params.max_tokens.max(1)`) while
                // skipping scheduler bookkeeping for the never-allocated
                // slot.
                let admitted_handle: Option<SlotHandle> = admitted.handle;

                // Vision-aware generate (Phase 2c Task #17 / iter-98 +
                // Wedge-4d). Build borrowed `SoftTokenInjection<'_>` /
                // `DeepstackInjection<'_>` slices from the owned data we
                // received over the channel; the borrow lifetime is
                // bounded by this match arm so it can't outlive the
                // underlying buffers.
                let injections: Vec<SoftTokenInjection<'_>> = soft_tokens
                    .iter()
                    .map(|d| SoftTokenInjection {
                        range: d.range.clone(),
                        embeddings: &d.embeddings,
                    })
                    .collect();
                let ds_borrow: Option<crate::serve::forward_prefill::DeepstackInjection<'_>> =
                    deepstack
                        .as_ref()
                        .map(|d| crate::serve::forward_prefill::DeepstackInjection {
                            image_token_positions: d.image_token_positions.clone(),
                            chunks: d.chunks.iter().collect(),
                        });
                let result = match &mut loaded {
                    LoadedModel::Gemma(g) => {
                        // Gemma path doesn't consume deepstack / 3D
                        // positions (those are Qwen3-VL specific). Fall
                        // back to the legacy soft-token-only entry.
                        generate_once_with_soft_tokens(
                            g,
                            &prompt_tokens,
                            &injections,
                            &params,
                            registration.as_ref(),
                        )
                    }
                    // ADR-005 Phase 4 Wedge-4a (2026-05-01): closes the
                    // last `qwen35_not_implemented_err()` call site —
                    // vision-aware generate now routes through
                    // `Qwen35Model::forward_gpu_last_logits_with_soft_tokens`
                    // via `engine_qwen35::generate_qwen35_once_with_soft_tokens`.
                    // **Wedge-4d (this iter)**: when `deepstack` is `Some`,
                    // route through
                    // `engine_qwen35::generate_qwen35_once_with_soft_tokens_and_deepstack`
                    // which threads the per-LM-layer DeepStack chunks
                    // and the 3D-mRoPE positions through the LM forward.
                    LoadedModel::Qwen35(q) => {
                        if ds_borrow.is_some() || positions_flat.is_some() {
                            super::engine_qwen35::generate_qwen35_once_with_soft_tokens_and_deepstack(
                                q,
                                &prompt_tokens,
                                &injections,
                                ds_borrow.as_ref(),
                                positions_flat.as_deref(),
                                &params,
                                registration.as_ref(),
                            )
                        } else {
                            super::engine_qwen35::generate_qwen35_once_with_soft_tokens(
                                q,
                                &prompt_tokens,
                                &injections,
                                &params,
                                registration.as_ref(),
                            )
                        }
                    }
                    // iter-9b: vision-aware generate via
                    // `engine_qwen3vl::generate_qwen3vl_text_with_soft_tokens_once`.
                    // Currently rejects non-empty soft_tokens with an
                    // explicit error pointing at iter-9c (forward-path
                    // soft-token splice not yet implemented). DeepStack
                    // chunks ARE wired (forward Phase D) and the 3D-
                    // mRoPE positions are passed through to the
                    // forward call. This unblocks DeepStack-only image
                    // chats and surfaces a clear error for
                    // soft-token-required image chats until iter-9c.
                    LoadedModel::Qwen3VlText(q) => {
                        // Use caller-supplied positions when present;
                        // fall back to text-only axis-major positions
                        // for callers that didn't build any (e.g. the
                        // chat handler routing a text-only request
                        // through this arm by accident).
                        let synth_positions: Vec<i32>;
                        let pos_slice: &[i32] = if let Some(p) = positions_flat.as_deref() {
                            p
                        } else {
                            let n = prompt_tokens.len();
                            synth_positions = {
                                let mut flat = vec![0i32; 4 * n];
                                for axis in 0..4 {
                                    for t in 0..n {
                                        flat[axis * n + t] = t as i32;
                                    }
                                }
                                flat
                            };
                            &synth_positions
                        };
                        super::engine_qwen3vl::generate_qwen3vl_text_with_soft_tokens_once(
                            q,
                            &prompt_tokens,
                            &injections,
                            ds_borrow.as_ref(),
                            pos_slice,
                            &params,
                            registration.as_ref(),
                        )
                    }
                    LoadedModel::Deepseek4(d) => {
                        if !injections.is_empty() || ds_borrow.is_some() || positions_flat.is_some()
                        {
                            Err(anyhow::anyhow!(
                                "DeepSeek-V4 does not support multimodal soft-token or \
                                 DeepStack injections"
                            ))
                        } else {
                            super::engine_deepseek4::generate_once(
                                d,
                                &prompt_tokens,
                                &params,
                                registration.as_ref(),
                            )
                        }
                    }
                };

                // ADR-040 C2b post-pattern — same bookkeeping shape as
                // Request::Generate; the inner `generate_*_with_soft_tokens*`
                // body is unchanged so byte-equivalence holds.
                //
                // cfa-iter-C2.5 M1: skip scheduler bookkeeping when admit
                // was zero-budget (`handle.is_none()`).
                if let Some(handle) = admitted_handle {
                    scheduler.advance_after_prefill(handle, prompt_tokens.len() as u32);
                    if let Ok(ref gr) = result {
                        for _ in 0..gr.completion_tokens {
                            scheduler.advance_after_decode(handle);
                        }
                    }
                    scheduler.release(handle);
                }
                publish_stats(&scheduler, &scheduler_stats_snapshot);

                let _ = reply.send(result);
            }
            Request::KvSnapshot {
                layer_rank,
                range,
                reply,
            } => {
                // Phase B-dense.2 follow-up: read a slice of
                // `dense_kvs[layer_rank]` K/V bytes via direct
                // `MlxBuffer::as_slice::<u8>()` access. The worker is
                // the sole owner of `LoadedModel` so there's no
                // concurrent GPU write — the as_slice safety contract
                // (no in-flight GPU command buffer) is satisfied as
                // long as the prior request's GPU work has drained,
                // which is guaranteed by the FIFO drain order at this
                // point (we only see KvSnapshot after the previous
                // request's reply was sent).
                let result: Result<Option<KvSnapshotBytes>> = match &mut loaded {
                    LoadedModel::Gemma(g) => kv_snapshot_gemma(g, layer_rank, range),
                    // Qwen35 has no dense_kvs surface; KV-persist for
                    // hybrid models lands under B-hybrid.1.
                    LoadedModel::Qwen35(_) => Ok(None),
                    // iter-228a Qwen3-VL text MVP: no dense_kvs surface
                    // until iter-228b allocates the KV cache. Match the
                    // Qwen35 shape (return Ok(None)) so the KV-spill
                    // hook treats it as "no snapshot available" rather
                    // than an error.
                    LoadedModel::Qwen3VlText(_) => Ok(None),
                    LoadedModel::Deepseek4(_) => Ok(None),
                };
                let _ = reply.send(result);
            }
            Request::KvRestore {
                layer_rank,
                range,
                k_payload,
                v_payload,
                write_pos,
                reply,
            } => {
                let result = match &mut loaded {
                    LoadedModel::Gemma(g) => {
                        kv_restore_gemma(g, layer_rank, range, &k_payload, &v_payload, write_pos)
                    }
                    LoadedModel::Qwen35(_) => Err(anyhow::anyhow!(
                        "kv_restore: not supported on Qwen35 variant (hybrid \
                         KV state — see B-hybrid.1)"
                    )),
                    // iter-228a Qwen3-VL text MVP: same shape — no KV
                    // restore until iter-228b allocates the cache.
                    LoadedModel::Qwen3VlText(_) => Err(anyhow::anyhow!(
                        "kv_restore: not yet supported on Qwen3-VL text variant \
                         (iter-228a is load-only; KV cache allocation lands in iter-228b)"
                    )),
                    LoadedModel::Deepseek4(_) => Err(anyhow::anyhow!(
                        "kv_restore: dense-KV restore is not applicable to DeepSeek-V4's \
                         compressed/recurrent serving cache"
                    )),
                };
                let _ = reply.send(result);
            }
            // **Phase B-tq.4** — TQ-packed snapshot/restore worker
            // dispatch.  Mirror of `Request::KvSnapshot`/`KvRestore`
            // for the TurboQuant-active KV path.  Reads/writes via
            // `MlxModelWeights::tq_v2_*` (shipped at
            // `forward_mlx.rs:4667+` in commit b7e975d).
            Request::TqPackedKvSnapshot {
                layer_rank,
                range,
                bits_per_coord,
                flags,
                scale,
                reply,
            } => {
                let result: Result<(Vec<u8>, Vec<u8>)> = match &loaded {
                    LoadedModel::Gemma(g) => g
                        .weights
                        .tq_v2_snapshot_block(layer_rank, range, bits_per_coord, flags, scale)
                        .map_err(|e| anyhow::anyhow!("tq_v2_snapshot_block failed: {:?}", e)),
                    LoadedModel::Qwen35(_) => Err(anyhow::anyhow!(
                        "tq_packed_kv_snapshot: not supported on Qwen35 variant \
                         (TQ-active path is Gemma 4 only at this iter — see B-tq.4)"
                    )),
                    LoadedModel::Qwen3VlText(_) => Err(anyhow::anyhow!(
                        "tq_packed_kv_snapshot: not supported on Qwen3-VL text \
                         variant (iter-228 is load-only; TQ-active wiring deferred)"
                    )),
                    LoadedModel::Deepseek4(_) => Err(anyhow::anyhow!(
                        "tq_packed_kv_snapshot: not supported on DeepSeek-V4"
                    )),
                };
                let _ = reply.send(result);
            }
            Request::TqPackedKvRestore {
                layer_rank,
                range,
                bits_per_coord,
                k_payload,
                v_payload,
                reply,
            } => {
                let result: Result<()> = match &mut loaded {
                    LoadedModel::Gemma(g) => g
                        .weights
                        .tq_v2_restore_block(
                            layer_rank,
                            range,
                            bits_per_coord,
                            &k_payload,
                            &v_payload,
                        )
                        .map_err(|e| anyhow::anyhow!("tq_v2_restore_block failed: {:?}", e)),
                    LoadedModel::Qwen35(_) => Err(anyhow::anyhow!(
                        "tq_packed_kv_restore: not supported on Qwen35 variant"
                    )),
                    LoadedModel::Qwen3VlText(_) => Err(anyhow::anyhow!(
                        "tq_packed_kv_restore: not supported on Qwen3-VL text variant"
                    )),
                    LoadedModel::Deepseek4(_) => Err(anyhow::anyhow!(
                        "tq_packed_kv_restore: not supported on DeepSeek-V4"
                    )),
                };
                let _ = reply.send(result);
            }
            Request::PromptCacheSnapshot { reply } => {
                // ADR-017 Closure iter-5 / Phase E: serialize the
                // loaded model's prompt_cache into JSON bytes. Only
                // the Gemma variant has a `prompt_cache: PromptCache`
                // field (engine_qwen35 uses HybridPromptCache, a
                // different type — return Ok(None) for now; B-hybrid
                // would extend with a Hybrid variant of the
                // serializer).
                let result: Result<Option<Vec<u8>>> = match &loaded {
                    LoadedModel::Gemma(g) => Ok(
                        crate::serve::kv_persist::prompt_cache_persist::try_serialize(
                            &g.prompt_cache,
                        ),
                    ),
                    LoadedModel::Qwen35(_) => Ok(None),
                    LoadedModel::Qwen3VlText(_) => Ok(None),
                    LoadedModel::Deepseek4(_) => Ok(None),
                };
                let _ = reply.send(result);
            }
            Request::PromptCacheRestore { payload, reply } => {
                // ADR-017 Closure iter-5 / Phase E: deserialize the
                // JSON payload into a PromptCache and assign to
                // loaded.prompt_cache. The next request that matches
                // tokens+key will hit iter-96's full-equality replay
                // path. Failure modes (parse error, version mismatch,
                // unknown finish_reason) yield Err so the caller can
                // fall through to fresh prefill — no silent
                // corruption.
                let result = match &mut loaded {
                    LoadedModel::Gemma(g) => {
                        match crate::serve::kv_persist::prompt_cache_persist::try_deserialize(&payload) {
                            Some(cache) => {
                                g.prompt_cache = cache;
                                Ok(())
                            }
                            None => Err(anyhow::anyhow!(
                                "prompt_cache_restore: deserialize failed (parse error, version mismatch, or unknown finish_reason)"
                            )),
                        }
                    }
                    LoadedModel::Qwen35(_) => Err(anyhow::anyhow!(
                        "prompt_cache_restore: not yet supported on Qwen35 hybrid variant — see B-hybrid follow-up"
                    )),
                    LoadedModel::Qwen3VlText(_) => Err(anyhow::anyhow!(
                        "prompt_cache_restore: not yet supported on Qwen3-VL text variant"
                    )),
                    LoadedModel::Deepseek4(_) => Err(anyhow::anyhow!(
                        "prompt_cache_restore: use DeepSeek-V4's live exact-prefix cache; \
                         serialized prompt-cache restore is not supported"
                    )),
                };
                let _ = reply.send(result);
            }
            Request::Shutdown => {
                tracing::info!("hf2q-engine worker received Shutdown; exiting");
                break;
            }
        }
    }

    tracing::info!("hf2q-engine worker thread exited");
}

// ---------------------------------------------------------------------------
// Phase B-dense.2 follow-up — KV snapshot/restore worker helpers
// ---------------------------------------------------------------------------
//
// These run from inside the worker thread (sole owner of
// `MlxModelWeights`). They read/write `weights.dense_kvs[layer].k.as_slice`
// directly without going through forward_mlx.rs (which is fenced from
// edits). The byte format mirrors what `gemma4_dense.rs` already
// produces in its `read_kv_range_to_bytes` / `write_bytes_into_kv_range`
// helpers — head-major: `[nkv_heads, n_tokens, head_dim]`.

/// Snapshot bytes from `dense_kvs[layer_rank]` over `range`. Returns
/// `Ok(None)` when `dense_kvs` is `None` (no prefill yet); returns
/// `Ok(Some(...))` with K+V bytes + shape on success; returns `Err(...)`
/// only on layer-out-of-range or `as_slice` failure.
fn kv_snapshot_gemma(
    loaded: &GemmaLoadedModel,
    layer_rank: usize,
    range: std::ops::Range<u32>,
) -> Result<Option<KvSnapshotBytes>> {
    let weights = &loaded.weights;
    if layer_rank >= weights.layers.len() {
        anyhow::bail!(
            "kv_snapshot: layer_rank {} out of range (num_layers={})",
            layer_rank,
            weights.layers.len()
        );
    }
    let layer_spec = &weights.layers[layer_rank];
    let nkv = layer_spec.num_kv_heads;
    let hd = layer_spec.head_dim;
    let is_sliding = layer_spec.layer_type == crate::serve::config::LayerType::Sliding;

    let kvs = match weights.dense_kvs.as_ref() {
        Some(v) => v,
        None => return Ok(None),
    };
    if layer_rank >= kvs.len() {
        anyhow::bail!(
            "kv_snapshot: dense_kvs len {} less than layer_rank {}",
            kvs.len(),
            layer_rank,
        );
    }
    let layer = &kvs[layer_rank];
    let capacity = layer.capacity;
    if capacity == 0 {
        return Ok(None);
    }
    if range.end <= range.start {
        anyhow::bail!("kv_snapshot: empty range");
    }
    let n_tokens = (range.end - range.start) as usize;
    if n_tokens > capacity {
        anyhow::bail!(
            "kv_snapshot: range len {} exceeds capacity {}",
            n_tokens,
            capacity,
        );
    }

    let dtype = layer.k.dtype();
    let elem_bytes = dtype.size_of();
    let head_stride_bytes = capacity * hd * elem_bytes;
    let tok_chunk_bytes = hd * elem_bytes;

    let k_src: &[u8] = layer
        .k
        .as_slice::<u8>()
        .map_err(|e| anyhow::anyhow!("kv_snapshot: K as_slice failed: {e}"))?;
    let v_src: &[u8] = layer
        .v
        .as_slice::<u8>()
        .map_err(|e| anyhow::anyhow!("kv_snapshot: V as_slice failed: {e}"))?;
    let expected_total = nkv * head_stride_bytes;
    if k_src.len() < expected_total || v_src.len() < expected_total {
        anyhow::bail!(
            "kv_snapshot: backing buffer shorter than expected ({}/{} vs {})",
            k_src.len(),
            v_src.len(),
            expected_total,
        );
    }

    let mut k_out = Vec::with_capacity(nkv * n_tokens * tok_chunk_bytes);
    let mut v_out = Vec::with_capacity(nkv * n_tokens * tok_chunk_bytes);

    if is_sliding {
        for h in 0..nkv {
            let head_base = h * head_stride_bytes;
            for tok in range.start..range.end {
                let slot = (tok as usize) % capacity;
                let off = head_base + slot * tok_chunk_bytes;
                let end = off + tok_chunk_bytes;
                if end > k_src.len() || end > v_src.len() {
                    anyhow::bail!("kv_snapshot: slot OOB at h={h} tok={tok}");
                }
                k_out.extend_from_slice(&k_src[off..end]);
                v_out.extend_from_slice(&v_src[off..end]);
            }
        }
    } else {
        for h in 0..nkv {
            let head_base = h * head_stride_bytes;
            for tok in range.start..range.end {
                let slot = tok as usize;
                if slot >= capacity {
                    anyhow::bail!("kv_snapshot: linear slot {} >= capacity {}", slot, capacity,);
                }
                let off = head_base + slot * tok_chunk_bytes;
                let end = off + tok_chunk_bytes;
                if end > k_src.len() || end > v_src.len() {
                    anyhow::bail!("kv_snapshot: linear slot OOB at h={h} tok={tok}");
                }
                k_out.extend_from_slice(&k_src[off..end]);
                v_out.extend_from_slice(&v_src[off..end]);
            }
        }
    }

    // Sliding write_pos is not tracked on `DenseKvBuffers` itself in
    // hf2q today (the live decode loop tracks it implicitly via
    // `self.kv_caches[i].write_pos`). For the snapshot bridge we
    // expose the current `kv_caches[layer_rank].write_pos` if the
    // layer is sliding; full-attention layers return the sentinel.
    let write_pos: u32 = if is_sliding {
        if let Some(kvc) = weights.kv_caches.get(layer_rank) {
            (kvc.write_pos as u32).min(u32::MAX - 1)
        } else {
            0
        }
    } else {
        u32::MAX
    };

    Ok(Some(KvSnapshotBytes {
        k: k_out,
        v: v_out,
        nkv_heads: nkv as u16,
        head_dim: hd as u16,
        capacity: capacity as u32,
        is_sliding,
        write_pos,
    }))
}

/// Restore K/V bytes into `dense_kvs[layer_rank]` at the slot positions
/// implied by `range`. Allocates `dense_kvs` if `None` (mirroring
/// `forward_prefill.rs:274-285`).
fn kv_restore_gemma(
    loaded: &mut GemmaLoadedModel,
    layer_rank: usize,
    range: std::ops::Range<u32>,
    k_payload: &[u8],
    v_payload: &[u8],
    write_pos: u32,
) -> Result<()> {
    // Read shape from the layer spec FIRST; we'll need it for both
    // alloc and write paths.
    let weights = &mut loaded.weights;
    if layer_rank >= weights.layers.len() {
        anyhow::bail!(
            "kv_restore: layer_rank {} out of range (num_layers={})",
            layer_rank,
            weights.layers.len()
        );
    }
    let layer_spec = &weights.layers[layer_rank];
    let nkv = layer_spec.num_kv_heads;
    let hd = layer_spec.head_dim;
    let is_sliding = layer_spec.layer_type == crate::serve::config::LayerType::Sliding;
    let sliding_window = weights.sliding_window;

    if range.end <= range.start {
        anyhow::bail!("kv_restore: empty range");
    }
    let n_tokens = (range.end - range.start) as usize;

    // Allocate dense_kvs if needed. Mirror the prefill allocator's
    // shape: sliding capacity = sliding_window, full-attn capacity =
    // n_tokens (we use the payload's range as the seq_len hint;
    // forward_prefill will reallocate at its real seq_len when it
    // runs, which is the same pattern gemma4_dense.rs:374-403 uses).
    if weights.dense_kvs.is_none() {
        let dev = loaded.ctx.device();
        let num_layers = weights.layers.len();
        let mut all = Vec::with_capacity(num_layers);
        for li in 0..num_layers {
            let s_nkv = weights.layers[li].num_kv_heads;
            let s_hd = weights.layers[li].head_dim;
            let s_is_sliding =
                weights.layers[li].layer_type == crate::serve::config::LayerType::Sliding;
            let s_cap = if s_is_sliding {
                sliding_window
            } else {
                // Linear capacity: at least enough for the payload.
                // Use a generous default so the first prefill's
                // realloc doesn't truncate already-restored bytes.
                //
                // ADR-017 Closure iter-13 (2026-05-05) note: the
                // `n_tokens.max(512)` default is INTENTIONALLY small.
                // The spiller writes 256-token blocks with ranges
                // [0..256), [256..512), [512..768), ..., so for
                // prefills > 512 tokens the 3rd+ blocks BAIL with
                // "linear slot N >= capacity 512" → IoErr → spiller
                // skip-and-continue. This is OBSERVABLE as
                // `restore_block layer=N IoErr (skipped, continuing)`
                // log lines in R-P5 traces.
                //
                // **The bug is cosmetic, not correctness-affecting**:
                // forward_prefill at `forward_prefill.rs:1502`
                // unconditionally REPLACES `self.dense_kvs = Some(...)`
                // at the next request's first prefill — overwriting
                // ANY restored bytes regardless of whether the
                // restore loop completed successfully. So allocating
                // a buffer big enough to hold ALL spilled blocks
                // (32768 × 8 KV-heads × 256 head-dim × 4 bytes × 64
                // layers ≈ 16 GiB) just so the restore loop can
                // populate bytes that are immediately discarded
                // would be pure waste — and could OOM the M5 Max
                // when stacked against the 15 GB model weights.
                //
                // Phase E (full-equality PromptCache replay,
                // iter-5/6) is the path that produces the
                // measurable cache hit; KV-block restore is
                // PRESERVED AS SUBSTRATE for a future Phase E
                // option (a) LCP partial-prefill resume that
                // would actually CONSUME the restored KV bytes.
                // Until that lands, the IoErr-skip behavior is
                // working-as-intended best-effort restore.
                n_tokens.max(512)
            };
            // dtype: use the layer-K's allocator dtype convention by
            // reading INVESTIGATION_ENV.f16_kv (mirrors prefill at
            // forward_prefill.rs:259-260). All layers share dtype on
            // Gemma 4.
            let kv_dtype = if crate::debug::INVESTIGATION_ENV.f16_kv {
                mlx_native::DType::F16
            } else {
                mlx_native::DType::F32
            };
            let elem = kv_dtype.size_of();
            let nbytes = s_nkv * s_cap * s_hd * elem;
            let k = dev
                .alloc_buffer(nbytes, kv_dtype, vec![s_nkv, s_cap, s_hd])
                .map_err(|e| anyhow::anyhow!("kv_restore: alloc K layer {li} failed: {e}"))?;
            let v = dev
                .alloc_buffer(nbytes, kv_dtype, vec![s_nkv, s_cap, s_hd])
                .map_err(|e| anyhow::anyhow!("kv_restore: alloc V layer {li} failed: {e}"))?;
            all.push(crate::inference::models::gemma4::DenseKvBuffers {
                k,
                v,
                capacity: s_cap,
                is_sliding: s_is_sliding,
                // ADR-017 Phase E.a iter-3.5a — dtype invariant.
                // `kv_dtype` was set above from
                // `INVESTIGATION_ENV.f16_kv`, same value the spiller
                // passed in via the snapshot's k/v alloc_buffer
                // parameter — stays in lockstep.
                dtype: kv_dtype,
            });
        }
        // ADR-017 Phase E.a iter-2.5: wrap each freshly-allocated
        // `DenseKvBuffers` in an Arc to match the field's
        // `Option<Vec<Arc<DenseKvBuffers>>>` shape. The just-built
        // `all` Vec is consumed; each entry's strong_count starts at 1
        // (no LcpRegistry clone yet — that's iter-3 territory).
        weights.dense_kvs = Some(all.into_iter().map(Arc::new).collect());
    }

    let kvs = weights
        .dense_kvs
        .as_mut()
        .ok_or_else(|| anyhow::anyhow!("kv_restore: dense_kvs alloc failed silently"))?;
    if layer_rank >= kvs.len() {
        anyhow::bail!(
            "kv_restore: dense_kvs len {} less than layer_rank {}",
            kvs.len(),
            layer_rank,
        );
    }
    // ADR-017 Phase E.a iter-2.5: at iter-2.5 the LcpRegistry holds
    // only marker payload `()`, so the Arc-cloned strong_count for
    // every per-layer entry is 1 (worker thread is sole holder).
    // `Arc::get_mut` always returns `Some(&mut DenseKvBuffers)` here.
    // Iter-3 will need a Cow-on-write or registry-handoff discipline
    // when the registry holds Arc-clones of the same buffer; until
    // then the unwrap is safe by construction.
    let layer = Arc::get_mut(&mut kvs[layer_rank]).ok_or_else(|| {
        anyhow::anyhow!(
            "kv_restore (iter-2.5): dense_kvs[layer_rank={}] Arc not exclusive — \
             LcpRegistry must hold marker payload `()` (no Arc<DenseKvBuffers> \
             clones outstanding) until iter-3 wires the partial-prefill resume path",
            layer_rank,
        )
    })?;
    let capacity = layer.capacity;
    if capacity == 0 {
        anyhow::bail!("kv_restore: zero capacity layer");
    }
    if is_sliding && capacity != sliding_window {
        anyhow::bail!(
            "kv_restore: sliding layer capacity {} != sliding_window {}",
            capacity,
            sliding_window,
        );
    }

    let dtype = layer.k.dtype();
    let elem_bytes = dtype.size_of();
    let head_stride_bytes = capacity * hd * elem_bytes;
    let tok_chunk_bytes = hd * elem_bytes;
    let expected_payload_bytes = nkv * n_tokens * tok_chunk_bytes;
    if k_payload.len() != expected_payload_bytes || v_payload.len() != expected_payload_bytes {
        anyhow::bail!(
            "kv_restore: payload size mismatch (k={}, v={}, expected={})",
            k_payload.len(),
            v_payload.len(),
            expected_payload_bytes,
        );
    }

    let k_dst: &mut [u8] = layer
        .k
        .as_mut_slice::<u8>()
        .map_err(|e| anyhow::anyhow!("kv_restore: K as_mut_slice failed: {e}"))?;
    let dst_total = nkv * head_stride_bytes;
    if k_dst.len() < dst_total {
        anyhow::bail!(
            "kv_restore: K backing buffer too small ({} < {})",
            k_dst.len(),
            dst_total,
        );
    }
    if is_sliding {
        let mut payload_off = 0usize;
        for h in 0..nkv {
            let head_base = h * head_stride_bytes;
            for tok in range.start..range.end {
                let slot = (tok as usize) % capacity;
                let off = head_base + slot * tok_chunk_bytes;
                let end = off + tok_chunk_bytes;
                if end > k_dst.len() {
                    anyhow::bail!("kv_restore: K slot OOB sliding");
                }
                k_dst[off..end]
                    .copy_from_slice(&k_payload[payload_off..payload_off + tok_chunk_bytes]);
                payload_off += tok_chunk_bytes;
            }
        }
    } else {
        let mut payload_off = 0usize;
        for h in 0..nkv {
            let head_base = h * head_stride_bytes;
            for tok in range.start..range.end {
                let slot = tok as usize;
                if slot >= capacity {
                    anyhow::bail!("kv_restore: linear slot {} >= capacity {}", slot, capacity,);
                }
                let off = head_base + slot * tok_chunk_bytes;
                let end = off + tok_chunk_bytes;
                if end > k_dst.len() {
                    anyhow::bail!("kv_restore: K slot OOB linear");
                }
                k_dst[off..end]
                    .copy_from_slice(&k_payload[payload_off..payload_off + tok_chunk_bytes]);
                payload_off += tok_chunk_bytes;
            }
        }
    }
    // Same loop again for V — separate scope so the &mut borrow on
    // layer.k drops before we re-borrow layer.v.
    let v_dst: &mut [u8] = layer
        .v
        .as_mut_slice::<u8>()
        .map_err(|e| anyhow::anyhow!("kv_restore: V as_mut_slice failed: {e}"))?;
    if v_dst.len() < dst_total {
        anyhow::bail!(
            "kv_restore: V backing buffer too small ({} < {})",
            v_dst.len(),
            dst_total,
        );
    }
    if is_sliding {
        let mut payload_off = 0usize;
        for h in 0..nkv {
            let head_base = h * head_stride_bytes;
            for tok in range.start..range.end {
                let slot = (tok as usize) % capacity;
                let off = head_base + slot * tok_chunk_bytes;
                let end = off + tok_chunk_bytes;
                v_dst[off..end]
                    .copy_from_slice(&v_payload[payload_off..payload_off + tok_chunk_bytes]);
                payload_off += tok_chunk_bytes;
            }
        }
    } else {
        let mut payload_off = 0usize;
        for h in 0..nkv {
            let head_base = h * head_stride_bytes;
            for tok in range.start..range.end {
                let slot = tok as usize;
                let off = head_base + slot * tok_chunk_bytes;
                let end = off + tok_chunk_bytes;
                v_dst[off..end]
                    .copy_from_slice(&v_payload[payload_off..payload_off + tok_chunk_bytes]);
                payload_off += tok_chunk_bytes;
            }
        }
    }

    // Restore sliding write_pos. For full-attention layers the
    // sentinel `u32::MAX` is ignored.
    if is_sliding && write_pos != u32::MAX {
        if let Some(kvc) = weights.kv_caches.get_mut(layer_rank) {
            kvc.write_pos = (write_pos as usize) % sliding_window.max(1);
        }
    }

    Ok(())
}

// ---------------------------------------------------------------------------
// Inference pipeline (synchronous, owned by the worker thread)
// ---------------------------------------------------------------------------

/// Single-pass warmup: run prefill + 1 decode on a tiny canary prompt to
/// compile all kernels and fault in the hot weights.
///
/// Iter-215 Wedge-2: takes `&mut GemmaLoadedModel` directly (after the
/// `LoadedModel` enum lift).  The Qwen35 variant has its own no-op
/// warmup path in the worker (the worker arm returns 501 immediately,
/// so warmup is effectively a no-op for that variant in MVP).
fn warmup_once(loaded: &mut GemmaLoadedModel) -> Result<()> {
    let started = Instant::now();
    // A 1-token prompt is enough to cycle through the prefill + decode path.
    // Use the GGUF bos-token id if available; else fall back to 1.
    let bos: u32 = 1;
    let prompt = vec![bos];
    let max_tokens = 1;
    let last_token = loaded
        .weights
        .forward_prefill(&prompt, max_tokens, &mut loaded.ctx)?;
    // One decode step to exercise the decode kernel set.
    let mut profiler = None;
    let _ =
        loaded
            .weights
            .forward_decode(last_token, prompt.len(), &mut loaded.ctx, &mut profiler)?;
    // Discard the warmup's per-prefill cache state.  warmup runs with
    // `prompt_len=1, max_tokens=1` → `linear_capacity = 2` allocated for
    // every per-layer KV buffer.  The `is_none()` re-alloc guards at
    // forward_prefill.rs:841 and forward_prefill_batched.rs:419 (load-
    // bearing for spec-decode multi-call cache reuse) would otherwise
    // make the FIRST real chat completion fail with
    // `cache_capacity(2)` inside the V-quantize / FA dispatch.  Mirrors
    // the embedding-mode reset at forward_prefill.rs:2216-2218 (same root
    // cause: a small-budget prefill poisons capacity for subsequent
    // calls).
    //
    // Codex review of `dbbd6009` (2026-05-17) flagged `dense_kvs_snapshot_for_lcp`
    // as ALSO needing clearing: with HF2Q_KV_LCP_RESUME=1 + HF2Q_USE_DENSE=1,
    // forward_prefill_with_soft_tokens_resume populates this snapshot and
    // the post-prefill store sites (engine.rs:5058, :7615) `take()` it
    // under the real prompt's LCP key.  A leftover BOS-sized warmup
    // snapshot would be installed against the FIRST real request's key.
    loaded.weights.dense_kvs = None;
    loaded.weights.dense_sdpa_tmp = None;
    loaded.weights.leg_hb_encoded = None;
    loaded.weights.hybrid_kv = None;
    loaded.weights.dense_kvs_snapshot_for_lcp = None;
    // "gemma-hybrid-lcp" (2026-08-03): same warmup-clearing discipline
    // for the hybrid leg snapshot (a leftover BOS-sized warmup snapshot
    // must never be installed against the first real request's key).
    loaded.weights.hybrid_kv_snapshot_for_lcp = None;
    tracing::info!(
        "hf2q-engine warmup complete in {:.0}ms",
        started.elapsed().as_secs_f64() * 1000.0
    );
    Ok(())
}

/// ADR-017 Phase E.a default-on + Codex Phase-2b audit (re-audit LOW issue
/// #2) — module-level auto-disable + warn-once helper for configurations
/// where LCP resume is default-ON but the active KV substrate has no
/// proven restore path.
///
/// **Resumable substrates (2026-08-03):**
/// - Gemma 4 dense (`HF2Q_USE_DENSE=1`) — the original iter-3 path.
/// - Gemma 4 hybrid (`HF2Q_HYBRID_KV` production default) — restored
///   per-layer dual-leg (dense + hybrid) payloads landed in
///   "gemma-hybrid-lcp" (see `GemmaLcpLayerKv`).
/// - Qwen 3.5/3.6 TQ-only — restored all four TQ buffers per slot in
///   ADR-027 sub-iter 23d-γ.
/// The HB-encoded opt-out regime (`hybrid_kv=0` without use_dense) has
/// NO restore path for its packed-K leg and stays auto-disabled.
///
/// The auto-disable preserves backward compatibility: operators on an
/// unproven substrate do not pay correctness or performance surprises
/// from the default flip.
///
/// Both the non-streaming (`generate_once_with_soft_tokens`) and the
/// streaming (`generate_stream_once`) probe sites call this helper.
/// The internal `std::sync::Once` guarantees exactly ONE log line per
/// process across both call sites.
///
/// **Effective LCP behavior** (computed at each call site, not here):
/// - `resumable_substrate=true` → LCP enabled regardless.
/// - `resumable_substrate=false && HF2Q_KV_LCP_RESUME` was explicitly `"1"` →
///   operator opt-in; LCP remains enabled (escape hatch; the existing
///   "misconfiguration" warning covers the mismatch).
/// - `resumable_substrate=false && default-on (env not explicitly "1")` →
///   auto-disable; this function fires once.
fn warn_lcp_resume_without_dense() {
    static ONCE: std::sync::Once = std::sync::Once::new();
    ONCE.call_once(|| {
        eprintln!(
            "[hf2q lcp] LCP partial-prefill resume is default-ON but the \
             active KV substrate has no proven restore path (dense \
             HF2Q_USE_DENSE=1 / gemma hybrid / qwen35-TQ are resumable; \
             HB-encoded opt-out is not); auto-disabling LCP for this \
             process. Set HF2Q_KV_LCP_RESUME=0 to silence."
        );
    });
}

/// Compute the *effective* `kv_lcp_resume` flag at a request gate site,
/// applying the Q3 auto-disable rule:
///
/// - If `parsed` is false → disabled (env was explicitly `=0`/`=off`/etc.).
/// - If `resumable_substrate` is true → enabled (see the substrate list
///   above; callers compute it per-arch).
/// - If `resumable_substrate` is false AND `HF2Q_KV_LCP_RESUME` was
///   explicitly `"1"` → operator override: remain enabled (warn once via
///   a different path).
/// - If `resumable_substrate` is false AND default-on (env not explicitly
///   `"1"`) → auto-disable and emit the warn-once.
///
/// Returns the effective bool.
pub(crate) fn effective_kv_lcp_resume(parsed: bool, resumable_substrate: bool) -> bool {
    if !parsed {
        return false;
    }
    if resumable_substrate {
        return true;
    }
    // Substrate unproven. Check if user explicitly set the var to "1".
    let explicitly_one = crate::debug::investigation_env::is_kv_lcp_resume_explicitly_one();
    if explicitly_one {
        // Operator intentionally set HF2Q_KV_LCP_RESUME=1 even on an
        // unproven substrate. Honour their intent; the existing
        // misconfig warning covers this.
        return true;
    }
    // Default-on + unproven substrate → auto-disable with a single warn-once.
    warn_lcp_resume_without_dense();
    false
}

/// ADR-017 Phase E option (a) iter-2 — build the per-request `LcpKey`
/// from a Gemma-loaded model + sampling params.
///
/// The fingerprint side reuses the SAME provenance recipe the spiller
/// uses (`Gemma4DenseSpill::model_fingerprint` at
/// `gemma4_dense.rs:1326-1333`) so that ANY prompt seen on the
/// KV-spill path and ANY prompt seen on the LCP-registry path key
/// against the SAME byte-stable `ModelFingerprint`. That coherence
/// matters at iter-3 wire-up time because the registry's payload will
/// be Arc-cloned `DenseKvBuffers` whose KV state was generated under
/// the same fingerprint — fingerprint divergence between the two
/// caches would mask cross-cache safety regressions.
///
/// The `tenant_id` is hardcoded to the empty string for v1 (single-
/// tenant). `params_hash = 0` because text-only Gemma 4 dense KV
/// state generation is independent of decode-time sampling params
/// (temperature, top_p, etc. apply post-prefill). Iter-3+ may tighten
/// this if grammar / soft_tokens / RoPE-affecting flags become
/// per-request configurable.
/// ADR-017 Phase E.a "gemma-hybrid-lcp" (2026-08-03) — zip the per-layer
/// dense + hybrid end-of-prefill snapshots into the `GemmaLcpLayerKv`
/// registry payload.
///
/// Returns `None` (skip store → clean cache miss, never fatal) when:
///   * the two snapshots disagree on layer count (producer bug — a
///     mismatched pairing would restore the wrong leg per layer), or
///   * any snapshot Arc is unexpectedly shared (`MlxBuffer` has no
///     Clone, so a contested Arc cannot be salvaged; skipping the store
///     keeps the registry honest instead of stashing a corrupt entry).
///     Exclusive-by-construction: both snapshots are minted during THIS
///     prefill and never published before this store, so contention is
///     a structural impossibility — this arm exists to fail safe, not
///     to handle a real case.
///
/// When `hybrid_snapshot` is `None` (dense / HB-encoded regimes), the
/// payload is dense-only `GemmaLcpLayerKv::Dense` — byte-identical to
/// the pre-sub-iter store behavior.
fn build_gemma_lcp_payload(
    dense_snapshot: Vec<std::sync::Arc<crate::inference::models::gemma4::DenseKvBuffers>>,
    hybrid_snapshot: Option<Vec<std::sync::Arc<crate::inference::models::gemma4::HybridKvBuffers>>>,
) -> Option<Vec<std::sync::Arc<crate::inference::models::gemma4::GemmaLcpLayerKv>>> {
    use crate::inference::models::gemma4::GemmaLcpLayerKv;
    match hybrid_snapshot {
        Some(hsnap) => {
            if hsnap.len() != dense_snapshot.len() {
                tracing::debug!(
                    "gemma-hybrid-lcp: dense/hybrid snapshot layer count mismatch \
                     ({} vs {}) — skipping store",
                    dense_snapshot.len(),
                    hsnap.len()
                );
                return None;
            }
            let mut out = Vec::with_capacity(dense_snapshot.len());
            for (idx, (d, h)) in dense_snapshot
                .into_iter()
                .zip(hsnap.into_iter())
                .enumerate()
            {
                let d = std::sync::Arc::try_unwrap(d)
                    .map_err(|arc| {
                        tracing::debug!(
                            "gemma-hybrid-lcp: dense snapshot Arc[{idx}] unexpectedly shared \
                             (strong_count={}) — skipping store",
                            std::sync::Arc::strong_count(&arc)
                        );
                    })
                    .ok()?;
                let h = std::sync::Arc::try_unwrap(h)
                    .map_err(|arc| {
                        tracing::debug!(
                            "gemma-hybrid-lcp: hybrid snapshot Arc[{idx}] unexpectedly shared \
                             (strong_count={}) — skipping store",
                            std::sync::Arc::strong_count(&arc)
                        );
                    })
                    .ok()?;
                out.push(std::sync::Arc::new(GemmaLcpLayerKv::DenseAndHybrid(d, h)));
            }
            Some(out)
        }
        None => {
            let mut out = Vec::with_capacity(dense_snapshot.len());
            for (idx, d) in dense_snapshot.into_iter().enumerate() {
                let d = std::sync::Arc::try_unwrap(d)
                    .map_err(|arc| {
                        tracing::debug!(
                            "gemma-hybrid-lcp: dense snapshot Arc[{idx}] unexpectedly shared \
                             (strong_count={}) — skipping store",
                            std::sync::Arc::strong_count(&arc)
                        );
                    })
                    .ok()?;
                out.push(std::sync::Arc::new(GemmaLcpLayerKv::Dense(d)));
            }
            Some(out)
        }
    }
}

fn build_lcp_key_for_request(
    loaded: &GemmaLoadedModel,
    _params: &SamplingParams,
) -> crate::serve::kv_persist::lcp_registry::LcpKey {
    use crate::serve::kv_persist::format::compute_model_fingerprint;
    let (producer_version, source_sha256) = match &loaded.provenance {
        crate::core::provenance::Provenance::Hf2q {
            producer_version,
            source_sha256,
            ..
        } => (producer_version.as_str(), source_sha256.as_str()),
        crate::core::provenance::Provenance::External => ("", ""),
    };
    // External provenance ⇒ chat-template hash empty (legacy
    // fallback at gemma4_dense.rs:1318-1324: External is
    // `(repo, quant, "", "", "")` — preserves pre-iter-211 namespace
    // collisions across re-quants of the same model).
    let chat_template_hash = match &loaded.provenance {
        crate::core::provenance::Provenance::Hf2q { .. } => {
            super::kv_spill_descriptor::KvSpillProvenance::hash_chat_template(&loaded.chat_template)
        }
        crate::core::provenance::Provenance::External => String::new(),
    };
    let quant = loaded.quant_type.as_deref().unwrap_or("");
    let fp = compute_model_fingerprint(
        &loaded.model_id,
        quant,
        producer_version,
        source_sha256,
        &chat_template_hash,
    );
    crate::serve::kv_persist::lcp_registry::LcpKey {
        model_fingerprint: fp,
        tenant_id: String::new(),
        params_hash: 0,
    }
}

/// Generate one full response: prefill the prompt, then decode up to
/// `max_tokens`, halting on EOS or a configured stop string. The decode path
/// is greedy-argmax (temperature 0). Richer sampling (top-p, top-k, seed,
/// logit_bias) lands when the grammar stack (Decision #6) comes in — the
/// sampler hook is the same.
fn generate_once(
    loaded: &mut GemmaLoadedModel,
    prompt_tokens: &[u32],
    params: &SamplingParams,
    registration: Option<&super::registry::ModelRegistration>,
) -> Result<GenerationResult> {
    generate_once_with_soft_tokens(loaded, prompt_tokens, &[], params, registration)
}

/// Vision-aware variant — same as `generate_once` except the prefill
/// goes through `forward_prefill_with_soft_tokens` so per-position
/// embedding overrides apply.  Phase 2c Task #17 / iter-98.
///
/// When `soft_tokens` is empty, behaviour is byte-identical to
/// `generate_once`.
fn generate_once_with_soft_tokens(
    loaded: &mut GemmaLoadedModel,
    prompt_tokens: &[u32],
    soft_tokens: &[SoftTokenInjection<'_>],
    params: &SamplingParams,
    registration: Option<&super::registry::ModelRegistration>,
) -> Result<GenerationResult> {
    anyhow::ensure!(
        !prompt_tokens.is_empty(),
        "generate_once: empty prompt_tokens"
    );
    let prompt_len = prompt_tokens.len();
    let max_tokens = params.max_tokens.max(1);

    // ── Prompt cache fast-path (Phase 2a Task #7 / iter-96) ────────────
    //
    // When the request is fully deterministic (greedy: T=0, no top_k /
    // top_p / repetition_penalty / seed) AND the prompt_tokens exactly
    // match the previous request's prompt, replay the cached result.
    // Skips the entire prefill+decode chain — the only cost is the
    // O(N) prompt-tokens equality compare.  The OpenAI usage shape
    // surfaces `cached_tokens = prompt_len` so clients can attribute
    // the saving.
    //
    // Sampling-mode bypasses the cache: replaying a deterministic
    // greedy decode for a sampling request would silently violate the
    // user's expectation of per-call variation.  See `PromptCache`
    // module doc for the full eligibility rules.
    if let Some(cached) = loaded.prompt_cache.lookup(prompt_tokens, params) {
        tracing::debug!(
            "prompt_cache: HIT — {} tokens served from cache, prefill+decode skipped",
            cached.prompt_tokens
        );
        return Ok(cached);
    }

    // ── ADR-017 Phase E option (a) — LCP partial-prefix probe ──
    //
    // Two layers (iter-2 observability + iter-3 env-gated resume):
    //
    //   1. iter-2: bump `hf2q_kv_lcp_lookups_total` always and
    //      `hf2q_kv_lcp_detected_total` when a non-trivial partial-
    //      prefix opportunity exists (`0 < K < N`). Multimodal request
    //      (`!soft_tokens.is_empty()`) ⇒ probe returns `None`
    //      unconditionally; the lookup counter still increments but
    //      the detected counter doesn't — operators reading /metrics
    //      can attribute the gap.
    //
    //   2. iter-3: when `HF2Q_KV_LCP_RESUME=1` AND `HF2Q_USE_DENSE=1`
    //      (TQ-packed kv_caches not safely resumable without a
    //      separate restoration path — Phase E.a v2 scope) AND probe
    //      hit AND capacity precondition holds (cached
    //      `linear_capacity` ≥ this request's `seq_len + max_tokens`,
    //      cached `sliding_window` matches model), CONSUME the cached
    //      Arc clones via `take_prefix`, install into
    //      `loaded.weights.dense_kvs`, and pass `Some(K)` to the
    //      partial-prefill resume entry point. Otherwise (any gate
    //      fails) the path falls back to the pre-iter-3 wholesale
    //      reset + fresh allocation.
    let resume_lcp: Option<usize> = {
        let lcp_key = build_lcp_key_for_request(loaded, params);
        let detected = crate::serve::kv_persist::lcp_registry::probe_lcp_opportunity(
            &mut loaded.lcp_registry,
            &lcp_key,
            prompt_tokens,
            !soft_tokens.is_empty(),
        );
        if let Some(sink) = loaded.kv_metrics_sink.as_ref() {
            sink.record_lcp_probe(detected);
        }
        match detected {
            None => None,
            Some(_k_obs) => {
                // Q3 auto-disable: compute effective LCP flag. Under default-on
                // with HF2Q_USE_DENSE=0, effective_kv_lcp_resume emits a
                // warn-once and returns false. Explicit HF2Q_KV_LCP_RESUME=1
                // overrides the auto-disable.
                // "gemma-hybrid-lcp" (2026-08-03): resumable substrates
                // = dense (HF2Q_USE_DENSE=1) OR production hybrid. The
                // HB-encoded opt-out regime stays auto-disabled.
                let lcp_enabled = effective_kv_lcp_resume(
                    crate::debug::INVESTIGATION_ENV.kv_lcp_resume,
                    crate::debug::INVESTIGATION_ENV.use_dense
                        || crate::debug::INVESTIGATION_ENV.hybrid_kv,
                );
                if !lcp_enabled {
                    None
                } else {
                    // Capacity check is a re-probe: take_prefix returns
                    // a fresh `LcpPrefix` (consuming the registry
                    // entry), and we cross-reference its capacities
                    // against this request's needs. A failed check
                    // bails to None AND re-stores nothing — the
                    // registry entry is already gone (consumed), but
                    // the post-prefill store path below will re-publish
                    // a fresh entry from this request's outputs, so
                    // future hits aren't permanently broken.
                    let prefix_opt = loaded.lcp_registry.take_prefix(&lcp_key, prompt_tokens);
                    match prefix_opt {
                        None => None,
                        Some(prefix) => {
                            // Aggregate capacity check.
                            let new_linear = prompt_tokens.len() + params.max_tokens.max(1);
                            let model_sw = loaded.weights.sliding_window.max(1);
                            let agg_ok = prefix.linear_capacity >= new_linear
                                && prefix.sliding_window == model_sw;
                            // Codex audit MED issue #2: per-layer
                            // cap + is_sliding check BEFORE installing
                            // the cached Arcs into weights. If any
                            // layer's cached capacity < required or
                            // is_sliding mismatches the model's layer
                            // type, fall through to fresh prefill
                            // GRACEFULLY (drop cached Arcs; engine
                            // alloc-fresh on the None path) instead of
                            // letting forward_prefill bail with a 500
                            // after the install side-effect.
                            let per_layer_ok = if !agg_ok {
                                false
                            } else if prefix.dense_kvs.len() != loaded.weights.layers.len() {
                                false
                            } else {
                                // ADR-017 Phase E.a iter-3.5a — dtype
                                // invariant added to the per-layer
                                // check. Model-current `kv_dtype` is
                                // resolved from `INVESTIGATION_ENV.f16_kv`
                                // (same source used at every alloc
                                // site). A cached entry with mismatched
                                // dtype must NOT be installed: the
                                // kernel's flash_attn_vec dispatch
                                // takes dtype as a static branch and
                                // would silently misread the cached
                                // bytes.
                                let model_kv_dtype = if crate::debug::INVESTIGATION_ENV.f16_kv {
                                    mlx_native::DType::F16
                                } else {
                                    mlx_native::DType::F32
                                };
                                // ADR-017 Phase E.a iter-3.6 follow-up
                                // (Codex audit LOW #1): align per-layer
                                // sliding required_cap with the alloc
                                // formula. When LONG_RESUME=1, sliding
                                // layers were allocated with
                                // `max(sw, new_linear)`; the per-layer
                                // check must demand ≥ same value, not
                                // just `model_sw`. Today the aggregate
                                // `prefix.linear_capacity >= new_linear`
                                // saves us, but a future refactor could
                                // admit an undersized sliding snapshot.
                                // "gemma-hybrid-lcp": long-resume admits
                                // dense OR production hybrid (kernel
                                // mask_type=2 verified for both legs).
                                let lr_long = crate::debug::INVESTIGATION_ENV.kv_lcp_long_resume
                                    && crate::debug::INVESTIGATION_ENV.kv_lcp_resume
                                    && (crate::debug::INVESTIGATION_ENV.use_dense
                                        || crate::debug::INVESTIGATION_ENV.hybrid_kv);
                                prefix.dense_kvs.iter().enumerate().all(|(li, arc)| {
                                    let layer = &loaded.weights.layers[li];
                                    let layer_is_ring = matches!(
                                        layer.layer_type,
                                        crate::serve::config::LayerType::Sliding
                                    );
                                    let required_cap = if layer_is_ring {
                                        if lr_long {
                                            model_sw.max(new_linear)
                                        } else {
                                            model_sw
                                        }
                                    } else {
                                        new_linear
                                    };
                                    // "gemma-hybrid-lcp" (2026-08-03):
                                    // the per-layer check runs on the
                                    // DENSE leg (prefill SDPA reads it);
                                    // the dense fields live behind
                                    // `arc.dense()` in the enum payload.
                                    let d = arc.dense();
                                    let dense_ok = d.capacity >= required_cap
                                        && d.is_sliding == layer_is_ring
                                        && d.dtype == model_kv_dtype;
                                    // Regime-consistency: under the
                                    // production hybrid regime the entry
                                    // MUST carry the hybrid leg per
                                    // layer — a dense-only entry under
                                    // hybrid would leave the decode cache
                                    // unrestored (silent zero-prefix; the
                                    // class this sub-iter exists to close).
                                    let regime_ok = if crate::debug::INVESTIGATION_ENV.hybrid_kv {
                                        match arc.hybrid() {
                                            Some(h) => {
                                                h.capacity >= required_cap
                                                    && h.is_sliding == layer_is_ring
                                            }
                                            None => false,
                                        }
                                    } else {
                                        true
                                    };
                                    dense_ok && regime_ok
                                })
                            };
                            if !per_layer_ok {
                                // Drop the cached Arcs (registry already
                                // consumed); fall through to fresh
                                // prefill. Log so operators can see
                                // capacity misses.
                                tracing::debug!(
                                    "lcp_resume: capacity check failed (agg_ok={}, \
                                     per_layer_ok={}, prefix.linear_cap={}, \
                                     new_linear={}, prefix.sw={}, model_sw={}) — \
                                     falling back to fresh prefill",
                                    agg_ok,
                                    per_layer_ok,
                                    prefix.linear_capacity,
                                    new_linear,
                                    prefix.sliding_window,
                                    model_sw,
                                );
                                drop(prefix);
                                None
                            } else {
                                let k = prefix.k;
                                // "gemma-hybrid-lcp" (2026-08-03): split the
                                // enum payload into the dense-leg Arc install
                                // (`weights.dense_kvs`, consumed by
                                // forward_prefill's restored_lcp branch) and
                                // the hybrid-leg OWNED install
                                // (`weights.hybrid_kv`, mutated in place by
                                // the per-token hybrid encode for positions
                                // [k..seq_len)). Arc::try_unwrap on the
                                // hybrid leg mirrors the dense path's
                                // exclusivity precondition (take_prefix
                                // leaves strong_count == 1); on violation we
                                // bail to fresh prefill GRACEFULLY (the
                                // capacity-fail branch's exact semantics —
                                // never a 500 from a cache hit).
                                let mut dense_arcs: Vec<
                                    std::sync::Arc<
                                        crate::inference::models::gemma4::DenseKvBuffers,
                                    >,
                                > = Vec::with_capacity(prefix.dense_kvs.len());
                                let mut hybrid_owned: Vec<
                                    crate::inference::models::gemma4::HybridKvBuffers,
                                > = Vec::new();
                                let mut install_ok = true;
                                for arc in prefix.dense_kvs.into_iter() {
                                    match std::sync::Arc::try_unwrap(arc) {
                                        Ok(layer) => match layer {
                                            crate::inference::models::gemma4::GemmaLcpLayerKv::Dense(
                                                d,
                                            ) => {
                                                dense_arcs.push(std::sync::Arc::new(d));
                                            }
                                            crate::inference::models::gemma4::GemmaLcpLayerKv::DenseAndHybrid(
                                                d,
                                                h,
                                            ) => {
                                                dense_arcs.push(std::sync::Arc::new(d));
                                                hybrid_owned.push(h);
                                            }
                                        },
                                        Err(arc) => {
                                            tracing::debug!(
                                                "gemma-hybrid-lcp: payload Arc unexpectedly \
                                                 shared at install (strong_count={}) — fresh prefill",
                                                std::sync::Arc::strong_count(&arc)
                                            );
                                            install_ok = false;
                                            break;
                                        }
                                    }
                                }
                                if !install_ok {
                                    drop(dense_arcs);
                                    drop(hybrid_owned);
                                    None
                                } else {
                                    let has_hybrid = !hybrid_owned.is_empty();
                                    // Install the per-layer Arcs into the
                                    // model. After this assignment, the
                                    // engine holds the only Arcs (registry
                                    // dropped its set in `take_prefix`,
                                    // strong_count == 1 per layer).
                                    loaded.weights.dense_kvs = Some(dense_arcs);
                                    if has_hybrid {
                                        loaded.weights.hybrid_kv = Some(hybrid_owned);
                                    }
                                    tracing::debug!(
                                        "lcp_resume: ENGAGED — K={} of N={} (per-layer cap ok)",
                                        k,
                                        prompt_tokens.len(),
                                    );
                                    Some(k)
                                }
                            }
                        }
                    }
                }
            }
        }
    };
    if let Some(k) = resume_lcp {
        tracing::debug!(
            "lcp_resume: dispatching forward_prefill_with_soft_tokens_resume(K={})",
            k
        );
    }

    // ── Sampler config — Tier 2/3/4 surface + grammar (iter-94 / iter-95) ──
    //
    // Pre-iter-94 the decode loop only consumed `forward_decode`'s
    // on-GPU greedy argmax — every `temperature` / `top_p` / `top_k` /
    // `repetition_penalty` / `logit_bias` request was silently downcast
    // to greedy.  Iter-94 forks on whether ANY field requests non-greedy
    // sampling and routes those through `sampler_pure::sample_token`
    // over the live `self.activations.logits` slice.  Iter-95 adds the
    // grammar branch: when `params.grammar.is_some()`, mask the live
    // logits via `grammar::mask::mask_invalid_tokens` BEFORE handing
    // them to `sampler_pure` (or to the greedy argmax for T=0).  The
    // chosen token's bytes then advance the runtime so the next step's
    // mask is correctly narrowed.
    //
    // Greedy fast path (all fields at default + no grammar) keeps the
    // existing forward_decode return-value chain — no logits readback,
    // no extra copy.  Sampling/grammar slow path discards the on-GPU
    // argmax token (~20 µs of wasted GPU work, negligible vs the
    // ~10-100ms layer forward) and re-derives the next token from the
    // mask + sample chain.
    //
    // ADR-020 AC#7 — `params.logprobs` ALSO forces the slow path so
    // we can read logits CPU-side + compute log_softmax(logits)[chosen]
    // via sampler_pure::sample_token_with_logprob.  Greedy GPU-argmax
    // skips the readback, so without this we have no logits over which
    // to compute the per-token logprob.
    let sample_logits = params.temperature > 0.0
        || params.top_k > 0
        || params.top_p < 1.0
        || params.repetition_penalty != 1.0
        || !params.logit_bias.is_empty()
        || params.grammar.is_some()
        || params.logprobs;
    let sampler_params = if sample_logits {
        Some(SamplerParams {
            temperature: params.temperature as f64,
            top_p: params.top_p as f64,
            top_k: params.top_k,
            min_p: 0.0,
            repetition_penalty: effective_repetition_penalty(params),
            max_tokens: params.max_tokens,
        })
    } else {
        None
    };

    // Build the per-request grammar runtime (Phase 2a Task #5 / iter-95).
    // `Grammar` is `Clone` (cheap ~Vec<Vec<GretElement>>); the runtime
    // owns the clone + a small Vec<Stack> of in-flight positions.  We
    // mutate in place across decode steps (advance via accept_bytes
    // after each sampled token).
    //
    // Wave 2.6 W-α5 Q2: when `params.grammar_kind == ToolCallBody`, the
    // runtime starts SUSPENDED via `set_awaiting_trigger(true)`.  The
    // mask + accept calls below are unconditional — the runtime
    // self-gates internally (mirrors llama.cpp lazy-grammar pattern at
    // /opt/llama.cpp/src/llama-grammar.cpp:1287-1344, citation in
    // research-report.md Q2).  The trigger flips when the
    // `ToolCallSplitter` sees the per-model open marker (handler
    // below).  For `GrammarKind::ResponseFormat` the runtime starts
    // EAGER — enforcement from token 0, byte-identical to pre-A1
    // behavior.  This is the wave-2.5 audit divergence A1 fix.
    let mut grammar_runtime: Option<super::grammar::GrammarRuntime> = match params.grammar.as_ref()
    {
        Some(g) => {
            let start_rule_id = g
                .rule_id("root")
                .ok_or_else(|| anyhow::anyhow!("grammar has no root rule"))?;
            let mut rt = super::grammar::GrammarRuntime::new(g.clone(), start_rule_id)
                .ok_or_else(|| anyhow::anyhow!("grammar runtime init failed"))?;
            // Wave 2.7 W-η Q-A: only `ToolCallBodyAuto` arms the lazy
            // (awaiting_trigger) gate.  `ToolCallBodyRequired` is EAGER
            // from token 0 — the grammar root already wraps the body in
            // open/close markers, so the mask must fire at byte 0 and
            // reject any token whose decoded bytes don't prefix the
            // open marker.  Mirrors llama.cpp `grammar_lazy = false` for
            // `tool_choice == REQUIRED` at common/chat.cpp:898-913,
            // 1177-1200, 1399-1416.
            if matches!(params.grammar_kind, GrammarKind::ToolCallBodyAuto) {
                rt.set_awaiting_trigger(true);
            }
            Some(rt)
        }
        None => None,
    };
    let token_bytes_ref: Option<&[Vec<u8>]> = params.token_bytes.as_deref().map(|v| &v[..]);

    // Wave-2.5 A1 / Wave 2.6 W-α5 Q2: ToolCallSplitter for the
    // non-streaming decode loop.  Used here ONLY to detect the per-model
    // open marker so we can call `runtime.trigger()` on the grammar — the
    // runtime then self-gates (no separate `in_body` boolean needed).
    // For `GrammarKind::ResponseFormat` runtimes the trigger is a no-op
    // because the runtime was constructed eager.  The splitter is `None`
    // when the model has no tool markers registered; in that case the
    // runtime never gets a trigger event but is also never suspended
    // (ResponseFormat default, or ToolCallBody on an unregistered model
    // which compile_tool_grammar refuses upstream).
    let mut tc_splitter_ns: Option<super::registry::ToolCallSplitter> =
        registration.and_then(|r| super::registry::ToolCallSplitter::from_registration(r));

    // Local helper — apply grammar mask + Tier 4 logit_bias and sample.
    // Mutably borrows the runtime so it can be advanced after sampling
    // (caller does the advance to keep this closure side-effect-light).
    // Returns the sampled token id; caller must feed
    // `token_bytes[id]` through the runtime to keep it in sync.
    //
    // Wave 2.6 W-α5 Q2: the mask call is UNCONDITIONAL.  The runtime
    // self-gates via `is_awaiting_trigger()` inside
    // `mask::mask_invalid_tokens` — when suspended (ToolCallBody
    // pre-trigger), the function early-returns 0 and leaves logits
    // untouched.  This removes the wave-2.5 `if in_tool_body { mask }`
    // wrapper and the sibling `Arc<AtomicBool>` it implied — exactly
    // the architecture the audit caught at engine.rs:1401, 1489, etc.
    // ADR-020 AC#7 — closure returns (token, optional logprob).
    // Logprob is `Some` iff the request set `logprobs:true`; computed
    // via `sampler_pure::sample_token_with_logprob` over the
    // post-bias / post-grammar-mask logits (so the logprob reflects
    // the distribution the sampler actually ran against).
    let want_logprobs = params.logprobs;
    let sample_from_live_logits = |weights: &mut MlxModelWeights,
                                   generated: &[u32],
                                   runtime: Option<&super::grammar::GrammarRuntime>|
     -> Result<(u32, Option<f32>)> {
        let sp = sampler_params.as_ref().expect("sample_logits gate");
        let mut logits: Vec<f32> = weights.logits_view()?.to_vec();
        // Tier 4 logit_bias FIRST: additive per OpenAI convention.
        if !params.logit_bias.is_empty() {
            let v = logits.len();
            for (&id, &bias) in &params.logit_bias {
                let idx = id as usize;
                if idx < v {
                    logits[idx] += bias;
                }
            }
        }
        // Grammar mask: zero out tokens that would drive the runtime
        // dead.  Self-gates on `runtime.is_awaiting_trigger()` —
        // suspended runtimes mask zero tokens (preamble freedom for
        // ToolCallBody-kind grammars before the open marker fires).
        if let (Some(rt), Some(tb)) = (runtime, token_bytes_ref) {
            super::grammar::mask::mask_invalid_tokens(rt, tb, &mut logits);
        }
        if want_logprobs {
            let (tok, lp) = sampler_pure::sample_token_with_logprob(&mut logits, sp, generated);
            Ok((tok, Some(lp)))
        } else {
            Ok((sampler_pure::sample_token(&mut logits, sp, generated), None))
        }
    };
    // ADR-020 AC#7 — per-completion-token logprob accumulator.
    // Length tracks `completion_tokens` and is moved into
    // `GenerationResult.logprobs` at end-of-decode.  Stays `None` when
    // the request did not opt in to logprobs.
    let mut logprobs_acc: Option<Vec<f32>> = if want_logprobs {
        Some(Vec::with_capacity(params.max_tokens))
    } else {
        None
    };

    // --- Prefill ---
    // Iter-98: route through forward_prefill_with_soft_tokens. Empty
    // soft_tokens slice is the no-op identity over forward_prefill —
    // text-only requests pay zero overhead.
    //
    // ADR-028 iter-415: serve HTTP path was historically per-token.
    // forward_prefill_batched (iter-344 default-on, iter-343 verified
    // coherent at pp3813 on gemma4-ara-2pass-APEX-Q5_K_M) is ~20-47×
    // faster.  Opt-in via HF2Q_SERVE_BATCHED_PREFILL=1; gated to
    // text-only (no soft tokens, no LCP resume) for safety.
    let prefill_start = Instant::now();
    // ADR-028 iter-421 default-flipped: per iter-326 operator REFRAME #2
    // ("default should have the best things on that provide the best
    // mantra-aligned outcome for users").  Phase 15 has been validated 4x:
    // iter-415 short prompts byte-identical, iter-416 multi-turn coherent,
    // iter-420 pp3.4K byte-identical, iter-421 long-decode/sampling/
    // streaming all robust.  Opt out via `HF2Q_SERVE_BATCHED_PREFILL=0`
    // / `=false` / `=off` (matches iter-326 q6_K_NR2 default-on pattern).
    // Tri-state (2026-08-03 auto-fallback): explicit =1 FORCES the
    // batched route (operator override); explicit =0/=false/=off forces
    // the linear route; UNSET = auto — engage batched only when this
    // request's O(n²) mask overhead fits the available-memory budget.
    // A 92K-token opencode first turn allocated ~120 GB transient on
    // 2026-08-03 and died in Metal with a command-buffer error — no
    // user should need to know BATCHED=0 exists.
    let serve_batched_env = std::env::var("HF2Q_SERVE_BATCHED_PREFILL").ok();
    let batched_allowed = match serve_batched_env.as_deref() {
        Some(v) => !matches!(v.to_ascii_lowercase().as_str(), "0" | "false" | "off"),
        None => {
            let viable = crate::serve::forward_prefill_batched::serve_batched_route_viable(
                prompt_tokens.len(),
                loaded.weights.num_attention_heads,
            );
            if !viable {
                eprintln!(
                    "[hf2q batched prefill] auto-fallback to linear route: \
                     seq_len={} O(n²) mask overhead exceeds the available- \
                     memory budget (force-on with HF2Q_SERVE_BATCHED_PREFILL=1)",
                    prompt_tokens.len()
                );
            }
            viable
        }
    };
    let use_batched_serve = soft_tokens.is_empty() && resume_lcp.is_none() && batched_allowed;
    let prefill_argmax = if use_batched_serve {
        loaded
            .weights
            .forward_prefill_batched(prompt_tokens, max_tokens, 0, &mut loaded.ctx)?
    } else {
        loaded.weights.forward_prefill_with_soft_tokens_resume(
            prompt_tokens,
            soft_tokens,
            max_tokens,
            &mut loaded.ctx,
            resume_lcp,
            false, // slot_aware=false (ADR-040 STEP-1b): legacy byte-equivalent
        )?
    };
    let prefill_duration = prefill_start.elapsed();

    // First decode token: greedy fast-path uses prefill's on-GPU argmax;
    // sampling path re-derives from prefill's live logits buffer (last
    // prompt-token's lm_head output) so the user-controlled temperature
    // applies to the very first generated token, not just decode-loop
    // tokens 2..N.  The greedy-fast-path skips logits readback entirely.
    //
    // Wave 2.6 W-α5 Q2: the mask + accept calls are UNCONDITIONAL.  For
    // `GrammarKind::ToolCallBodyAuto` the runtime is suspended
    // (`is_awaiting_trigger() == true`) so both calls are no-ops and the
    // first token is naturally unconstrained — the same behavior the
    // wave-2.5 explicit `in_body_first = false` short-circuit
    // produced, but achieved structurally via the runtime self-gate.
    // For `GrammarKind::ResponseFormat` and Wave 2.7 W-η Q-A's
    // `ToolCallBodyRequired` the runtime enforces from token 0
    // (response_format fixes audit divergence A1; Required eagerly
    // constrains the model to emit a tool call from byte 0).
    let mut next_token = if sample_logits {
        let (tok, lp) =
            sample_from_live_logits(&mut loaded.weights, &[], grammar_runtime.as_ref())?;
        if let (Some(acc), Some(lp_val)) = (logprobs_acc.as_mut(), lp) {
            acc.push(lp_val);
        }
        let tok = tok;
        // Feed the chosen token's bytes through the grammar runtime so
        // the next step's mask is correctly narrowed.  No-op when no
        // grammar OR when the runtime is awaiting trigger (suspended
        // runtime self-gates).  Empty token_bytes (special/unprintable)
        // is also skipped — accept_bytes on empty is a true no-op.
        if let (Some(rt), Some(tb)) = (grammar_runtime.as_mut(), token_bytes_ref) {
            let bytes = tb.get(tok as usize).map(|v| v.as_slice()).unwrap_or(&[]);
            if !bytes.is_empty() {
                rt.accept_bytes(bytes);
            }
        }
        tok
    } else {
        prefill_argmax
    };

    // --- Reasoning splitter + counter (Decision #21) ---
    // Feed each decoded fragment through a local ReasoningSplitter; count
    // tokens whose post-feed state is `in_reasoning`. Mirrors the streaming
    // path's accounting exactly so stream + non-stream usage agree.
    let mut splitter = registration
        .filter(|r| r.has_reasoning())
        .and_then(|r| super::registry::make_reasoning_splitter(r, params.reasoning_forced_open));
    let reasoning_enabled = splitter.is_some();
    let mut reasoning_token_count: usize = 0;

    // --- Decode loop ---
    let decode_start = Instant::now();
    let mut generated_tokens: Vec<u32> = Vec::with_capacity(max_tokens);
    generated_tokens.push(next_token);

    // ADR-017 Phase E.a iter-3 + Codex Phase-2b audit follow-up:
    // physical decode-side KV write counter. Each `forward_decode`
    // call writes exactly one position to `dense_kvs[*][pos %
    // capacity]`. Tracking this explicitly (vs deriving from
    // `generated_tokens.len()` post-pop) makes the sliding-ring wrap
    // guard's boundary check unambiguous: the guard at the LCP store
    // path uses `prompt_len + physical_decode_writes` to decide
    // whether the ring wrapped. Rationale: grammar-dead path POPs the
    // last generated token, but the corresponding `forward_decode`
    // call DID write KV. `completion_tokens` (post-pop) underrepresents
    // physical writes by 1 in that case; an explicit counter is
    // immune to that off-by-one ambiguity.
    let mut physical_decode_writes: usize = 0;

    let first_fragment = loaded
        .tokenizer
        .decode(&[next_token], false)
        .unwrap_or_default();
    let mut decoded_text = first_fragment.clone();
    if let Some(sp) = splitter.as_mut() {
        let _ = sp.feed(&first_fragment);
        if sp.in_reasoning() {
            reasoning_token_count += 1;
        }
    }
    // Wave 2.6 W-α5 Q2: feed first fragment through the tool-call
    // splitter; if it emits a `ToolCallOpen` event, trigger the grammar
    // runtime so subsequent tokens are constrained by the body grammar.
    // (Typically the first decoded token is never the open marker, but
    // this keeps the state machine correct for any edge case where the
    // chat template ends mid-marker.)  llama.cpp does NOT reset the
    // trigger on close — multi-call support comes from the grammar
    // shape `(call)+`.  See research-report.md Q2 anti-finding +
    // /opt/llama.cpp/docs/function-calling.md.
    if let Some(tcs) = tc_splitter_ns.as_mut() {
        let events = tcs.feed(&first_fragment);
        if let Some(rt) = grammar_runtime.as_mut() {
            if events
                .iter()
                .any(|e| matches!(e, super::registry::ToolCallEvent::ToolCallOpen))
            {
                rt.trigger();
            }
        }
    }

    let mut finish_reason: &'static str = "length";
    let mut profiler = ProfileAccumulator::new(0);

    // Early EOS check on the prefill-emitted first token.
    if loaded.eos_token_ids.contains(&next_token) {
        finish_reason = "stop";
    } else if hit_stop_string(&decoded_text, &params.stop_strings) {
        finish_reason = "stop";
    } else {
        for _ in 1..max_tokens {
            let pos = prompt_len + generated_tokens.len() - 1;
            let mut p = profiler.start_token();
            // forward_decode populates self.activations.logits as a
            // side-effect of its lm_head + softcap dispatch chain; the
            // returned u32 is the on-GPU greedy argmax (only used on the
            // greedy fast-path).
            let greedy_token =
                loaded
                    .weights
                    .forward_decode(next_token, pos, &mut loaded.ctx, &mut p)?;
            // ADR-017 Phase E.a iter-3 — count physical KV write.
            // `forward_decode` always writes exactly one position; this
            // increments BEFORE any later EOS / stop_string / grammar-
            // dead branches that might pop or break, so the count
            // reflects actual GPU writes.
            physical_decode_writes += 1;
            profiler.finish_token(p);

            next_token = if sample_logits {
                // Sampling slow path: read logits, apply Tier 4 logit_bias,
                // grammar mask, then call sampler_pure for
                // temperature/top_p/top_k/rep-penalty.
                //
                // Wave 2.6 W-α5 Q2: mask + accept calls are
                // UNCONDITIONAL.  The runtime self-gates via
                // `is_awaiting_trigger()`; suspended runtimes (lazy
                // tool-call body grammar pre-trigger) mask zero tokens
                // and ignore advance, so preamble emission is naturally
                // unconstrained.  Eager runtimes (ResponseFormat)
                // enforce every step.  This removes the wave-2.5 sibling
                // `Arc<AtomicBool>` and the `if in_body { mask }` /
                // `if in_body { accept }` split that the audit caught at
                // engine.rs:1401, 1489.
                let (tok, lp) = sample_from_live_logits(
                    &mut loaded.weights,
                    &generated_tokens,
                    grammar_runtime.as_ref(),
                )?;
                if let (Some(acc), Some(lp_val)) = (logprobs_acc.as_mut(), lp) {
                    acc.push(lp_val);
                }
                // Advance the grammar runtime by the chosen token's bytes.
                // Self-gates internally — see GrammarRuntime::accept_bytes.
                if let (Some(rt), Some(tb)) = (grammar_runtime.as_mut(), token_bytes_ref) {
                    let bytes = tb.get(tok as usize).map(|v| v.as_slice()).unwrap_or(&[]);
                    if !bytes.is_empty() {
                        rt.accept_bytes(bytes);
                    }
                }
                tok
            } else {
                // Greedy fast path — use forward_decode's on-GPU argmax.
                greedy_token
            };

            if loaded.eos_token_ids.contains(&next_token) {
                finish_reason = "stop";
                break;
            }

            generated_tokens.push(next_token);
            let fragment = loaded
                .tokenizer
                .decode(&[next_token], false)
                .unwrap_or_default();
            decoded_text.push_str(&fragment);
            if let Some(sp) = splitter.as_mut() {
                let _ = sp.feed(&fragment);
                if sp.in_reasoning() {
                    reasoning_token_count += 1;
                }
            }
            // Wave 2.6 W-α5 Q2: feed the splitter; if it emits a
            // ToolCallOpen on this fragment, trigger the grammar runtime
            // so subsequent decode steps enforce the body grammar.
            // ToolCallClose does NOT reset the trigger — single-call
            // termination is delivered structurally by the grammar shape
            // exhausting after `body <tool_call|> space` (the iter-218
            // default `parallel_tool_calls=false` matches llama.cpp's
            // bounded `(call){min,max=1}` per
            // `/opt/llama.cpp/docs/function-calling.md:24`); multi-call
            // mode (`parallel_tool_calls=true` opt-in) carries via the
            // `gemma4-call*` shape which permits another open marker.
            if let Some(tcs) = tc_splitter_ns.as_mut() {
                let events = tcs.feed(&fragment);
                if let Some(rt) = grammar_runtime.as_mut() {
                    if events
                        .iter()
                        .any(|e| matches!(e, super::registry::ToolCallEvent::ToolCallOpen))
                    {
                        rt.trigger();
                    }
                }
            }
            if hit_stop_string(&decoded_text, &params.stop_strings) {
                finish_reason = "stop";
                // Strip the stop string from the returned text (OpenAI
                // convention per ADR-005 "Stop-sequence stripping from
                // returned text").
                strip_trailing_stop(&mut decoded_text, &params.stop_strings);
                break;
            }
            // Grammar-driven termination (Phase 2a Task #5 / iter-95).
            //
            // After the grammar runtime tried to accept the chosen
            // token, `is_dead()` becomes true if no in-flight stack
            // can extend further.  Two ways this fires after a
            // grammar-constrained decode step:
            //
            //   1. **Mask masked everything**: every printable token's
            //      bytes failed the grammar, so `sampler_pure` softmaxed
            //      all-`-inf` logits, summed to zero, and fell back to
            //      `indexed[0]` (usually id=0 = `<pad>` for Gemma).
            //      That token's bytes also fail the grammar (`<pad>`
            //      decodes to literal `"<pad>"` text — `<` is not valid
            //      JSON after `} ws`).  `accept_bytes` returned false
            //      above ⇒ runtime is now dead.
            //   2. **Grammar fully matched + last token was the final
            //      legal one**: the runtime accepted the token but has
            //      no remaining stacks — the parse is complete.
            //
            // Both cases collapse to "decoder should halt".  Pop the
            // last pushed token + re-decode the surviving prefix so
            // any out-of-grammar fragment (`<pad>`) doesn't appear in
            // the response body.
            if grammar_runtime.as_ref().is_some_and(|rt| rt.is_dead()) {
                finish_reason = "stop";
                generated_tokens.pop();
                decoded_text = loaded
                    .tokenizer
                    .decode(&generated_tokens, false)
                    .unwrap_or_default();
                break;
            }
        }
    }
    let decode_duration = decode_start.elapsed();

    // When finish_reason == "stop" but the EOS was seen, make sure the EOS
    // token text isn't present in the returned content.
    let _ = params; // params.temperature etc. are greedy defaults in this iter

    // Apply reasoning split (Decision #21) if this model has boundary
    // markers registered. If not, the full decoded text goes into
    // `content` and `reasoning_text` is `None`.
    let (content, reasoning_text) = match registration {
        Some(reg) if reg.has_reasoning() => super::registry::split_full_output_forced(
            reg,
            &decoded_text,
            params.reasoning_forced_open,
        ),
        _ => (decoded_text, None),
    };

    let result = GenerationResult {
        text: content,
        reasoning_text,
        prompt_tokens: prompt_len,
        completion_tokens: generated_tokens.len(),
        reasoning_tokens: if reasoning_enabled && reasoning_token_count > 0 {
            Some(reasoning_token_count)
        } else {
            None
        },
        finish_reason,
        prefill_duration,
        decode_duration,
        cached_tokens: 0, // iter-96: 0 on cache miss; > 0 on hit (handled by fast-path return earlier)
        logprobs: logprobs_acc,
    };

    // Store this generation in the prompt cache — same eligibility
    // gate as `lookup` (sampling-mode requests are not cached).  The
    // store happens AFTER all error paths above so a partial / failed
    // generation can never poison the cache.
    loaded.prompt_cache.store(prompt_tokens, params, &result);

    // ADR-017 Phase E option (a) iter-3 — record this prompt's
    // post-prefill KV state in the LCP registry so future requests
    // with shared-prefix prompts can be detected (iter-2 metric) and,
    // when env-gated ON, resumed via in-place reuse (iter-3).
    //
    // Iter-3 swaps iter-2's marker payload `()` for the real
    // `Vec<Arc<DenseKvBuffers>>` from `loaded.weights.dense_kvs`. We
    // KEEP one set of Arc clones in `weights` (so forward_decode can
    // continue reading from them) AND store another set in the
    // registry. Per-layer strong_count after this is 2 (engine +
    // registry); a future hit's `take_prefix` brings the count back
    // to 1 in the caller (registry drops its set), enabling the
    // partial-prefill resume path's `Arc::try_unwrap`.
    //
    // ## Sliding-ring wrap safety (Codex audit issue #1, R12)
    //
    // Decode mutates `dense_kvs[*][slot=p%sw]` for sliding layers as
    // it advances positions [N..N+M). When N+M > sliding_window, the
    // ring WRAPS: decode-written slots overwrite prompt-written slots
    // [0..(N+M-sw)). On a future LCP resume at K ≤ N, slots [0..K)
    // would no longer hold pure prompt prefix — they'd hold
    // assistant decode tokens. The kernel's permutation-invariant
    // sliding semantic (`forward_prefill.rs:466-471`) is violated for
    // resume because resume needs the slots to represent specific
    // positions, not "the most recent sw positions".
    //
    // V1 fix: gate the store on `prompt_tokens.len() +
    // physical_decode_writes <= sliding_window` (Codex re-audit
    // 2026-05-05: explicit physical-write counter incremented per
    // `forward_decode` call, immune to grammar-pop / EOS-break /
    // stop_string off-by-one accounting that `result.completion_tokens`
    // would have introduced). If decode wrapped (or would have
    // wrapped), don't store — the cached state is no
    // longer a faithful representation of the prompt prefix. This
    // makes long-conversation caching miss (each turn's prompt grows
    // and eventually exceeds sw) but preserves byte-identity
    // correctness, which is the load-bearing iter-3 v1 invariant.
    // Iter-3 v2 can lift this restriction by snapshotting dense_kvs
    // at end-of-prefill (before decode mutates) — adds ~5 GB GPU
    // memcpy per request on Gemma 4 26B; deferred for v1.
    //
    // Skip multimodal requests (`!soft_tokens.is_empty()`): the
    // text-only-bound LCP registry must not record prompts whose
    // KV state was generated under per-position soft-token overrides
    // (cf. dossier §10.5 multimodal bail).
    //
    // Skip when `loaded.weights.dense_kvs` is None — that happens
    // only on the embedding-only path (`forward_prefill_embedding`)
    // where dense_kvs is never built. Generation requests always
    // populate it.
    // ADR-017 Phase E.a iter-3.5b — store the END-OF-PREFILL SNAPSHOT
    // (NOT the live post-decode dense_kvs).
    //
    // forward_prefill_with_soft_tokens_resume populates
    // `loaded.weights.dense_kvs_snapshot_for_lcp = Some(snapshot)` at
    // end-of-prefill (BEFORE decode mutates) when the iter-3 env-gates
    // are on. Decode then mutates `loaded.weights.dense_kvs` (the
    // LIVE set) without touching the snapshot. Storing the snapshot
    // here gives future LCP hits a buffer that faithfully represents
    // [0..N) of the prompt — no decode-corrupted ring slots.
    //
    // The snapshot lifts the iter-3 v1 wrap restriction. Long-
    // conversation prompts where `prompt_len + decode_tokens >
    // sliding_window` are now cacheable; the wrap guard is GONE.
    //
    // Skip multimodal requests (`!soft_tokens.is_empty()`) per
    // dossier §10.5.
    //
    // Skip when `dense_kvs_snapshot_for_lcp` is None — that happens
    // when env-gates are off (no snapshot was taken; iter-2
    // observability-only mode), OR on the embedding-only path where
    // forward_prefill_embedding doesn't populate it.
    //
    // ADR-017 Phase E.a iter-3.5c — sliding-layer prefill-wrap guard.
    //
    // The end-of-prefill snapshot fixes DECODE-WRAP (decode mutates
    // live buffers; snapshot is taken before decode runs). It does
    // NOT fix PREFILL-WRAP: when `prompt_len > sliding_window`, the
    // sliding ring wraps DURING prefill itself. The snapshot then
    // captures the FINAL ring state — slots representing positions
    // `[N-sw..N)`, not `[0..N)`. A future LCP resume at K<N would
    // expect slots to represent `[0..K)` (P's shared prefix tokens);
    // mismatch corrupts P's attention output.
    //
    // The dossier §3.4 argues sliding LCP > sw is "safe" but its
    // argument assumes the cache was stored AT position LCP (mid-
    // prefill); my impl stores at end-of-Q's-prefill. Different
    // states. Per mantra "Never trust comments over code"; the code
    // says "skip store on prefill wrap until iter-3.6 implements
    // mid-prefill snapshot or rotated-ring resume".
    //
    // Skip the LCP store when the model has any sliding layer AND
    // `prompt_len > sliding_window`. Pure-dense (global-only) models
    // have linear-capacity buffers (max_position_embeddings, ~262144
    // for Gemma 4) that don't wrap; for those, prefill-wrap doesn't
    // exist. v1 limitation: long-prompt LCP hits skipped when sliding
    // layers are present. Multi-turn chat with prompts ≤ sw still
    // benefits.
    if soft_tokens.is_empty() {
        // "gemma-hybrid-lcp" (2026-08-03): take the hybrid leg snapshot
        // alongside the dense one; both are populated at end-of-prefill
        // under the production hybrid regime (None otherwise).
        let hybrid_snapshot = loaded.weights.hybrid_kv_snapshot_for_lcp.take();
        if let Some(snapshot) = loaded.weights.dense_kvs_snapshot_for_lcp.take() {
            // "gemma-hybrid-lcp": build the regime-aware payload. On
            // fail-safe (layer mismatch / shared Arc) this is None and
            // the store below is skipped (clean future miss, never fatal).
            let payload = build_gemma_lcp_payload(snapshot, hybrid_snapshot);
            let sliding_window = loaded.weights.sliding_window.max(1);
            let has_sliding_layer = loaded
                .weights
                .layers
                .iter()
                .any(|l| matches!(l.layer_type, crate::serve::config::LayerType::Sliding));
            // iter-3.5c prefill-wrap guard — distinct from iter-3 v1's
            // decode-wrap guard (which iter-3.5b removed).
            //
            // ADR-017 Phase E.a iter-3.6: when HF2Q_KV_LCP_LONG_RESUME=1,
            // sliding layers were allocated with linear (non-wrapping)
            // capacity in forward_prefill, so the snapshot captures
            // positions [0..N) faithfully even when N > sw. The guard
            // skip is no longer needed; lift it for the long-resume
            // path. (Default OFF: behavior is byte-identical to iter-7.)
            // "gemma-hybrid-lcp": long-resume admits dense OR production
            // hybrid (mirrors the probe-side gate).
            let kv_lcp_long_resume = crate::debug::INVESTIGATION_ENV.kv_lcp_long_resume
                && crate::debug::INVESTIGATION_ENV.kv_lcp_resume
                && (crate::debug::INVESTIGATION_ENV.use_dense
                    || crate::debug::INVESTIGATION_ENV.hybrid_kv);
            let prefill_safe =
                !has_sliding_layer || prompt_len <= sliding_window || kv_lcp_long_resume;
            // The `physical_decode_writes` counter is no longer
            // load-bearing for the (decode-)wrap guard (snapshot
            // makes the iter-3 v1 guard unnecessary); kept as a
            // debug-only counter.
            let _ = physical_decode_writes;
            if prefill_safe {
                if let Some(payload) = payload {
                    let lcp_key = build_lcp_key_for_request(loaded, params);
                    // ADR-017 Phase E.a iter-3.5d — multi-turn chat
                    // headroom. Snapshot global-layer buffers were
                    // allocated with capacity = sliding_window (not
                    // prompt_len + max_decode_tokens). Report the
                    // larger value here so the probe-side capacity check
                    // admits future turns whose prompts grow toward sw.
                    let linear_capacity = sliding_window.max(prompt_len + params.max_tokens.max(1));
                    match loaded.lcp_registry.store(
                        lcp_key,
                        prompt_tokens.to_vec(),
                        payload,
                        sliding_window,
                        linear_capacity,
                    ) {
                        Ok(()) => {}
                        Err(e) => {
                            tracing::debug!("lcp_registry.store rejected (unexpected): {:?}", e);
                        }
                    }
                } // if let Some(payload)
            } else {
                tracing::debug!(
                    "lcp_registry.store skipped: prefill-wrap guard \
                     (prompt_len={} > sliding_window={}; iter-3.5c \
                     correctness preserves byte-identity for sliding \
                     layers — iter-3.6 mid-prefill snapshot lifts this)",
                    prompt_len,
                    sliding_window
                );
            }
        }
    }

    Ok(result)
}

/// **ADR-040 iter-B4c-kernel iter-1 (2026-05-30)** — slot-aware
/// Gemma 4 chat-generation orchestrator that routes the worker hot
/// path through the persistent multi-seq per-layer
/// [`crate::inference::models::gemma4::kv_cache::MultiSeqHbKvBuffers`]
/// scaffold (`GemmaLoadedModel.multi_seq_kv`) instead of the legacy
/// per-request inline alloc.
///
/// Cross-architecture mirror of Qwen35 iter-C2d-cont-kernel iter-1
/// `engine_qwen35::generate_qwen35_once_slot_aware` per §6.1.27 — same
/// dispatch fork shape (`slot_id != SlotId(0)` predicate at the worker
/// arm), same take-and-restore borrow pattern, same `reset_for_slot`
/// entry+exit discipline.
///
/// **Scope (iter-1 only — Generate arm, scaffold-shape)**.  Per the
/// §6.1.31 closure block's path-decision discipline, this iter ships
/// the **structural primitives** (per-layer `reset_for_slot` calls +
/// take-and-restore borrow + dispatch fork) WITHOUT the kernel-level
/// `forward_prefill.rs` slot-offset routing.  The kernel-forward step
/// itself surfaces a typed `MultiSeqError::CapabilityUnsupported`
/// naming `iter-B4c-kernel-iter-2` as the implementing sub-iter — the
/// honest pin that the kernel work is NOT yet done.  This is structurally
/// distinct from Qwen35 iter-C2d-cont-kernel iter-1 which lifted a fully
/// `slot_id`-threaded `forward_gpu_last_logits` (B4b §6.1.20 had landed
/// the kernel-side slot threading on Qwen35); Gemma 4 has NO equivalent
/// of B4b — the kernel slot-offset routing through
/// `forward_prefill.rs:843-882` / `forward_prefill_batched.rs:443-475`
/// / `forward_gpu.rs:443-459` (the 3 inline alloc sites per §6.1.25)
/// is itself the staged `iter-B4c-kernel-iter-2` work.
///
/// The other 3 Gemma 4 worker arms (`Request::GenerateStream`,
/// `Request::Embed`, `Request::GenerateWithSoftTokens`) STILL carry
/// the C2c §6.1.21 typed `MultiSeqError::CapabilityUnsupported` clamp
/// with relabeled `iter-B4c-kernel-iter-{3,4,5}` deferral cites.  See
/// §6.1.31 for the iter-1 → iter-{2,3,4,5} sequencing decision.
///
/// **What iter-1 ships** (load-bearing primitives reused by future
/// sub-iters):
/// 1. The take-and-restore borrow pattern at the worker arm site
///    (`g.multi_seq_kv.take()` → call → `g.multi_seq_kv = Some(buf)`)
///    that resolves the partial-borrow conflict between
///    `&mut g.multi_seq_kv` and the dense `&mut g.lcp_registry` /
///    `&mut g.prompt_cache` accesses.
/// 2. Per-layer `MultiSeqHbKvBuffers::reset_for_slot(slot_id)` invocation
///    at entry + exit for cross-request isolation within the slot — the
///    new primitive added to `gemma4/kv_cache.rs` by this iter.
/// 3. The dispatch-fork shape (the worker arm distinguishes
///    `LoadedModel::Gemma(g)` SlotId(0) from SlotId(N>0) at the same
///    site Qwen35 distinguishes its surface).
/// 4. Defense-in-depth typed-error on the impossible
///    `multi_seq_kv.is_none()` branch at SlotId(N>0) (pinned by H81).
/// 5. The structural witness that `iter-B4c-kernel-iter-2` (the kernel
///    forward step) is the NAMED next iter — operator + reviewer +
///    future-iter grep'able cite via `MultiSeqError::CapabilityUnsupported`.
///
/// **What iter-1 does NOT ship** (typed sub-deferrals):
/// - `iter-B4c-kernel-iter-2`: the kernel-side `forward_prefill.rs` /
///   `forward_prefill_batched.rs` / `forward_gpu.rs` slot-offset routing
///   refactor — threading `slot_id: SlotId` through the 3 inline alloc
///   sites and the `dispatch_hadamard_quantize_kv_hb_*` callers per
///   §6.1.25 followup pointer.  Same primitive Qwen35 B4a-cont uses for
///   F32 KV slot-offset routing per §6.1.20.
/// - `iter-B4c-kernel-iter-3`: GenerateStream slot-aware orchestrator
///   (mirror of Qwen35 iter-C2d-cont-kernel iter-2 §6.1.28).
/// - `iter-B4c-kernel-iter-4`: Embed slot-aware orchestrator (mirror of
///   Qwen35 iter-C2d-cont-kernel iter-3 §6.1.29).
/// - `iter-B4c-kernel-iter-5`: GenerateWithSoftTokens slot-aware
///   orchestrator (mirror of Qwen35 iter-C2d-cont-kernel iter-4 §6.1.30).
/// - `iter-B4c-kernel-iter-LCP` / `iter-B4c-kernel-iter-G`: orthogonal
///   slot-aware LCP + greedy fast-path optimizations (parallel to the
///   Qwen35 sub-deferrals per §6.1.27).
///
/// **Per-slot byte-equivalence at SlotId(0)** (H77 pin): the worker
/// arm's `handle.slot_id != SlotId(0)` predicate short-circuits AT the
/// worker arm — `generate_gemma4_once_slot_aware` is NEVER called for
/// SlotId(0).  Both SerialFifo (always SlotId(0)) and SlotAware +
/// SlotId(0) (first-slot byte-equivalence pin) route through the
/// existing `generate_once` dispatch verbatim, preserving the H1/H2/H23
/// /H44 byte-equivalence chain that A5*/C2a/C2b/C2c/B4c closed.
///
/// **Investigation findings** (Gemma 4 kernel-prerequisite gap):
///
/// 1. **Qwen35 B4b precedent absent**: per §6.1.20, B4b shipped the
///    decode-path `slot_id` threading on Qwen35 (`forward_gpu_last_logits`
///    et al accept `SlotId`).  Gemma 4 has NO equivalent — `forward_prefill`
///    / `forward_prefill_with_soft_tokens` / `forward_embed_last` (all
///    in `src/serve/forward_prefill.rs`) have NO `slot_id` parameter
///    anywhere in their signatures or call graph.  Source-grep confirms:
///    `grep slot_id src/serve/forward_prefill.rs` → 0 hits.
/// 2. **3 inline alloc sites per §6.1.25**: `forward_prefill.rs:843-882`
///    (the `leg_hb_encoded` per-layer 3-D `HbKvBuffers` alloc loop),
///    `forward_prefill_batched.rs:443-475` (mirror for the batched path),
///    `forward_gpu.rs:443-459` (decode-path mirror).  Each currently
///    constructs the legacy 3-D `[nkv, capacity, head_dim]` shape; the
///    A3a `MultiSeqHbKvBuffers` outermost `n_seqs` axis lift is NOT
///    consulted anywhere in the forward path.
/// 3. **Scope verdict**: a Path A "full kernel lift bundled in iter-1"
///    requires the §6.1.25 §-cited LOC delta (>600 LOC across 3 files +
///    30 layers × 3 KV variants × xlen optional).  Per the brief's
///    permission for sub-iter scope reduction, iter-1 ships the
///    orchestrator scaffold + the structural reset_for_slot primitives
///    that the iter-2 kernel-forward step will consume; iter-2 lands
///    the actual kernel-side slot-offset routing.
///
/// # Errors
/// - `prompt_tokens.is_empty()`.
/// - `slot_id.0 >= multi_seq_kv[0].n_seqs` (via `reset_for_slot` bounds-
///   first per A2b §6.1.23 iter-1.5 cfa-finding-F5).
/// - `iter-B4c-kernel-iter-2` typed `CapabilityUnsupported` on the
///   kernel-forward step (the load-bearing pin until iter-2 lands).
fn generate_gemma4_once_slot_aware(
    loaded: &mut GemmaLoadedModel,
    prompt_tokens: &[u32],
    params: &SamplingParams,
    // ADR-040 iter-B4c-kernel iter-2-decode-C (2026-05-30) — lifted from
    // `_registration` to `registration`.  iter-2-decode-A's greedy fast-
    // path body never engaged the reasoning splitter / tool-call splitter,
    // so the param was prefixed with `_` to silence dead-code warnings.
    // iter-2-decode-C wires `ReasoningSplitter` + `ToolCallSplitter` so
    // reasoning-mode + tool-call requests at SlotId(N>0) route correctly;
    // both helpers take an `Option<&ModelRegistration>`.
    registration: Option<&super::registry::ModelRegistration>,
    multi_seq_kv: &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqHbKvBuffers>,
    // ADR-040 iter-B4c-kernel iter-2B (2026-05-30) — production-default
    // hybrid F16-K + TQ-HB-V scaffold sibling param.  `Option<>` because
    // iter-C2c-cont provisions this field IFF `INVESTIGATION_ENV.hybrid_kv
    // == true` (DEFAULT since ADR-029 iter-13 per H10 falsification at
    // §6.1.11); when the env-gate is OFF the worker arm passes `None` and
    // the new model fn defense-in-depth-fails if the hybrid branch is
    // reached.  Threaded verbatim through to the model fn — same shape
    // as the HB scaffold but for the production-default regime.
    //
    // `mut` binding so the orchestrator body can do entry+exit
    // `reset_for_slot` via `if let Some(ref mut _) = multi_seq_kv_hybrid`
    // AND pass an `as_deref_mut()` reborrow to the model fn call below.
    mut multi_seq_kv_hybrid: Option<
        &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqHybridKvBuffers>,
    >,
    // ADR-040 iter-B4c-kernel iter-2D (2026-05-30) — dense F32 scaffold
    // sibling param.  Provisioned IFF HF2Q_USE_DENSE=1 at SlotAware
    // spawn time (iter-C2c-cont-cont Phase 3, §6.1.46).  When None,
    // the iter-2D dispatch-fork branch in the model fn surfaces typed
    // `iter-C2c-cont-cont-invariant-violated` defense-in-depth.
    mut multi_seq_kv_dense: Option<
        &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqDenseKvBuffers>,
    >,
    // ADR-040 iter-B4c-kernel iter-2C (2026-05-30) — legacy 4-bit
    // nibble-packed scaffold sibling param.  Provisioned IFF
    // HF2Q_TQ_CODEBOOK_BITS=4 at SlotAware spawn time (iter-C2c-cont-cont
    // Phase 4, §6.1.46).  When None, the iter-2C dispatch-fork branch
    // in the model fn surfaces typed
    // `iter-C2c-cont-cont-invariant-violated` defense-in-depth.
    mut multi_seq_kv_mlx: Option<
        &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqMlxKvCache>,
    >,
    slot_id: SlotId,
) -> Result<GenerationResult> {
    anyhow::ensure!(
        !prompt_tokens.is_empty(),
        "generate_gemma4_once_slot_aware: empty prompt_tokens"
    );
    anyhow::ensure!(
        !multi_seq_kv.is_empty(),
        "generate_gemma4_once_slot_aware: multi_seq_kv is empty \
         (C2c spawn-arm invariant: provision_multi_seq_kv_for_slot_aware \
          must produce one entry per layer; ADR-040 §6.1.21)"
    );
    // Bounds-first per A2b §6.1.23 iter-1.5 cfa-finding-F5 ordering.
    // Use the first layer's n_seqs as the canonical bound — A3a
    // construction guarantees every per-layer entry has the same
    // n_seqs (provisioned with max_slots).
    let n_seqs = multi_seq_kv[0].n_seqs;
    anyhow::ensure!(
        slot_id.0 < n_seqs,
        "generate_gemma4_once_slot_aware: SlotOutOfRange slot={} max_slots={} \
         (ADR-040 iter-B4c-kernel iter-1)",
        slot_id.0,
        n_seqs,
    );

    // Per-slot reset at entry — the persistent cache may carry stale
    // bytes from a prior request on this slot.  `reset_for_slot` zeros
    // the per-seq cursor for `slot_id` on EVERY per-layer entry; K/V
    // packed + norms bytes are cursor-masked (see layout proof at
    // `MultiSeqHbKvBuffers::reset_for_slot`).
    //
    // Mirror of Qwen35 iter-C2d-cont-kernel iter-1 entry-reset
    // discipline per §6.1.27.
    for (layer_idx, buf) in multi_seq_kv.iter_mut().enumerate() {
        buf.reset_for_slot(slot_id).map_err(|e| {
            anyhow::anyhow!(
                "generate_gemma4_once_slot_aware: reset_for_slot at entry L{layer_idx}: {e} \
             (ADR-040 iter-B4c-kernel iter-1)"
            )
        })?;
    }
    // ADR-040 iter-B4c-kernel iter-2B (2026-05-30) — entry reset on the
    // hybrid scaffold sibling.  Mirrors the HB scaffold entry-reset
    // discipline above for the production-default regime.  Uses
    // `.as_deref_mut()` to reborrow so the model fn call below can
    // re-take the same `Option<&mut Vec<_>>` shape (the borrow
    // ends with this for-loop scope).
    if let Some(ref mut hybrid_scaffold) = multi_seq_kv_hybrid {
        for (layer_idx, buf) in hybrid_scaffold.iter_mut().enumerate() {
            buf.reset_for_slot(slot_id).map_err(|e| {
                anyhow::anyhow!(
                    "generate_gemma4_once_slot_aware: reset_for_slot at entry (hybrid) \
                 L{layer_idx}: {e} (ADR-040 iter-B4c-kernel iter-2B)"
                )
            })?;
        }
    }

    // ADR-040 iter-B4c-kernel iter-2A (2026-05-30) — kernel-forward
    // call lands.  Replaces iter-1's IIFE-wrapped typed
    // `CapabilityUnsupported` at the orchestrator boundary with a real
    // call into the load-bearing
    // `MlxModelWeights::forward_prefill_with_soft_tokens_slot_aware`
    // primitive (defined in `src/serve/forward_prefill.rs`).
    //
    // iter-2A advances the typed-deferral by ONE call-graph hop: the
    // model fn signature now ACCEPTS `slot_id: SlotId` + `&mut Vec<
    // MultiSeqHbKvBuffers>`, with a bounds-first pre-flight + per-
    // regime dispatch fork.  Each of the 4 production KV regimes
    // (hybrid F16-K + TQ-HB-V; HB-encoded; legacy 4-bit; dense F32)
    // surfaces its own named iter-2{A-cont,B,C,D} typed sub-deferral
    // — iter-2A-cont is the in-scope kernel-dispatch refactor (HB-
    // encoded path); iter-2B is the production-default (HF2Q_HYBRID_KV
    // =1 since ADR-029 iter-13 per H10 falsification at §6.1.11); iter-
    // 2C is the legacy 4-bit opt-in surface; iter-2D is the dense F32
    // LCP-eligible regime.
    //
    // Defense-in-depth: the call site reads `max_decode_tokens` from
    // `params.max_tokens.max(1)` (mirrors `generate_once` at line 6653)
    // so the per-layer KV alloc-sizing in the body inherits the same
    // capacity discipline as the SerialFifo path.  `&[]` soft_tokens
    // matches the Generate-arm path; the SoftTokens-arm port is iter-
    // B4c-kernel-iter-5.
    let max_decode_tokens = params.max_tokens.max(1);
    // ADR-040 iter-B4c-kernel iter-2B (2026-05-30) — production-default
    // hybrid scaffold threaded through to the model fn via
    // `.as_deref_mut()` reborrow (so we can re-borrow for the exit-reset
    // below this call without consuming the outer `Option<&mut Vec<_>>`).
    // ADR-040 iter-B4c-kernel iter-2-decode-A (2026-05-30) — Generate-arm
    // decode-loop body landed (greedy fast-path).  iter-2B's prefill
    // returns the first decode token via
    // `forward_prefill_with_soft_tokens_slot_aware`; iter-2-decode-A
    // calls `forward_decode_slot_aware` per token in a greedy fast-path
    // loop until EOS or `max_tokens`.
    //
    // ADR-040 iter-B4c-kernel iter-2-decode-C (2026-05-30) — FULL sampler/
    // grammar/stop-strings/logprobs/reasoning-text surface lands.
    // Mirror of `generate_once`'s slow path at engine.rs:7427-7896 with
    // the slot-aware kernel calls (`forward_prefill_with_soft_tokens_
    // slot_aware` + `forward_decode_slot_aware`) substituted for the
    // sibling fn calls.  Greedy fast-path (T=0, no grammar, no
    // logprobs, no stop_strings) is byte-equivalent to the
    // iter-2-decode-A landing (sampler skipped, logits readback
    // skipped); non-greedy path engages `sampler_pure::sample_token`
    // (or `sample_token_with_logprob`) over the live logits buffer
    // from `loaded.weights.logits_view()`.
    let kernel_forward_result: Result<GenerationResult> = (|| -> Result<GenerationResult> {
        let prefill_started = Instant::now();
        let first_decode_token = loaded.weights.forward_prefill_with_soft_tokens_slot_aware(
            prompt_tokens,
            &[], // Generate-arm: no soft-token overrides; vision-aware
            // path is iter-B4c-kernel-iter-5 (SoftTokens arm).
            max_decode_tokens,
            &mut loaded.ctx,
            slot_id,
            multi_seq_kv,
            multi_seq_kv_hybrid.as_deref_mut(),
            // ADR-040 iter-2D + iter-2C (§6.1.46): dense F32 + legacy
            // 4-bit scaffold siblings.  Threaded as Option<&mut> per
            // the iter-2B precedent — None when the respective env
            // gate is off (the model fn defense-in-depth-fails).
            multi_seq_kv_dense.as_deref_mut(),
            multi_seq_kv_mlx.as_deref_mut(),
        )?;
        let prefill_duration = prefill_started.elapsed();

        // ── Sampler / grammar / logprobs config ────────────────────────
        //
        // iter-2-decode-C: full surface mirror of `generate_once` at
        // engine.rs:7453-7585.  `sample_logits` predicate is the union
        // of every non-greedy field — when ANY is engaged, the slow
        // path reads logits CPU-side via `logits_view()`, applies Tier
        // 4 `logit_bias`, masks via the grammar runtime (if any), and
        // calls `sampler_pure::sample_token` / `sample_token_with_logprob`.
        let sample_logits = params.temperature > 0.0
            || params.top_k > 0
            || params.top_p < 1.0
            || params.repetition_penalty != 1.0
            || !params.logit_bias.is_empty()
            || params.grammar.is_some()
            || params.logprobs;
        let sampler_params = if sample_logits {
            Some(SamplerParams {
                temperature: params.temperature as f64,
                top_p: params.top_p as f64,
                top_k: params.top_k,
                min_p: 0.0,
                repetition_penalty: effective_repetition_penalty(params),
                max_tokens: params.max_tokens,
            })
        } else {
            None
        };

        // Grammar runtime — mirror of generate_once at engine.rs:7489-7510.
        // Lazy-trigger semantics for ToolCallBodyAuto preserved verbatim.
        let mut grammar_runtime: Option<super::grammar::GrammarRuntime> =
            match params.grammar.as_ref() {
                Some(g) => {
                    let start_rule_id = g
                        .rule_id("root")
                        .ok_or_else(|| anyhow::anyhow!("grammar has no root rule"))?;
                    let mut rt = super::grammar::GrammarRuntime::new(g.clone(), start_rule_id)
                        .ok_or_else(|| anyhow::anyhow!("grammar runtime init failed"))?;
                    if matches!(params.grammar_kind, GrammarKind::ToolCallBodyAuto) {
                        rt.set_awaiting_trigger(true);
                    }
                    Some(rt)
                }
                None => None,
            };
        let token_bytes_ref: Option<&[Vec<u8>]> = params.token_bytes.as_deref().map(|v| &v[..]);

        // Tool-call splitter for trigger detection.  Mirror of
        // engine.rs:7523-7524.  We use it for the lazy-grammar trigger;
        // there's no SSE channel here so the open-marker just flips the
        // grammar runtime out of awaiting_trigger mode.
        let mut tc_splitter_ns: Option<super::registry::ToolCallSplitter> =
            registration.and_then(super::registry::ToolCallSplitter::from_registration);

        // Per-completion-token logprob accumulator.
        let want_logprobs = params.logprobs;
        let mut logprobs_acc: Option<Vec<f32>> = if want_logprobs {
            Some(Vec::with_capacity(params.max_tokens))
        } else {
            None
        };

        // First decode token: greedy fast-path reuses prefill's on-GPU
        // argmax; sampling path re-derives from the live logits buffer
        // (last prompt-token's lm_head output) so user-controlled
        // temperature applies to the very first generated token.
        // Mirror of generate_once at engine.rs:7645-7665.
        let mut next_token = if sample_logits {
            let sp = sampler_params.as_ref().expect("sample_logits gate");
            let mut logits: Vec<f32> = loaded.weights.logits_view()?.to_vec();
            if !params.logit_bias.is_empty() {
                let v = logits.len();
                for (&id, &bias) in &params.logit_bias {
                    let idx = id as usize;
                    if idx < v {
                        logits[idx] += bias;
                    }
                }
            }
            if let (Some(rt), Some(tb)) = (grammar_runtime.as_ref(), token_bytes_ref) {
                super::grammar::mask::mask_invalid_tokens(rt, tb, &mut logits);
            }
            let (tok, lp_opt) = if want_logprobs {
                let (t, lp) = sampler_pure::sample_token_with_logprob(&mut logits, sp, &[]);
                (t, Some(lp))
            } else {
                (sampler_pure::sample_token(&mut logits, sp, &[]), None)
            };
            if let (Some(acc), Some(lp_val)) = (logprobs_acc.as_mut(), lp_opt) {
                acc.push(lp_val);
            }
            if let (Some(rt), Some(tb)) = (grammar_runtime.as_mut(), token_bytes_ref) {
                let bytes = tb.get(tok as usize).map(|v| v.as_slice()).unwrap_or(&[]);
                if !bytes.is_empty() {
                    rt.accept_bytes(bytes);
                }
            }
            tok
        } else {
            first_decode_token
        };

        // Reasoning splitter — classifies the running text; counter is
        // accumulated and exposed via GenerationResult.reasoning_tokens.
        // Mirror of generate_once at engine.rs:7671-7675.
        let mut splitter = registration.filter(|r| r.has_reasoning()).and_then(|r| {
            super::registry::make_reasoning_splitter(r, params.reasoning_forced_open)
        });
        let reasoning_enabled = splitter.is_some();
        let mut reasoning_token_count: usize = 0;

        let decode_started = Instant::now();
        let mut generated_tokens: Vec<u32> = Vec::with_capacity(max_decode_tokens);
        let mut decoded_text = String::new();

        // Decode the first emitted token into text BEFORE the EOS check.
        let first_fragment = loaded
            .tokenizer
            .decode(&[next_token], false)
            .unwrap_or_default();
        decoded_text.push_str(&first_fragment);
        if let Some(sp) = splitter.as_mut() {
            let _ = sp.feed(&first_fragment);
            if sp.in_reasoning() {
                reasoning_token_count += 1;
            }
        }
        if let Some(tcs) = tc_splitter_ns.as_mut() {
            let events = tcs.feed(&first_fragment);
            if let Some(rt) = grammar_runtime.as_mut() {
                if events
                    .iter()
                    .any(|e| matches!(e, super::registry::ToolCallEvent::ToolCallOpen))
                {
                    rt.trigger();
                }
            }
        }

        let mut finish_reason: &'static str = "length";

        // Early EOS / stop_string check on the prefill-emitted first
        // token (mirror of generate_once at engine.rs:7728-7732).
        if loaded.eos_token_ids.contains(&next_token) {
            finish_reason = "stop";
        } else if hit_stop_string(&decoded_text, &params.stop_strings) {
            finish_reason = "stop";
            strip_trailing_stop(&mut decoded_text, &params.stop_strings);
        } else {
            generated_tokens.push(next_token);
            for _ in 1..max_decode_tokens {
                let pos = prompt_tokens.len() + generated_tokens.len() - 1;
                let mut p: Option<crate::inference::models::gemma4::profile::TokenProfile> = None;
                let greedy_token = loaded.weights.forward_decode_slot_aware(
                    next_token,
                    pos,
                    &mut loaded.ctx,
                    &mut p,
                    slot_id,
                    multi_seq_kv,
                    multi_seq_kv_hybrid.as_deref_mut(),
                    // ADR-040 iter-2-decode-D (§6.1.46) — dense F32 +
                    // legacy 4-bit decode-side scaffold siblings.
                    multi_seq_kv_dense.as_deref_mut(),
                    multi_seq_kv_mlx.as_deref_mut(),
                )?;

                next_token = if sample_logits {
                    let sp = sampler_params.as_ref().expect("sample_logits gate");
                    let mut logits: Vec<f32> = loaded.weights.logits_view()?.to_vec();
                    if !params.logit_bias.is_empty() {
                        let v = logits.len();
                        for (&id, &bias) in &params.logit_bias {
                            let idx = id as usize;
                            if idx < v {
                                logits[idx] += bias;
                            }
                        }
                    }
                    if let (Some(rt), Some(tb)) = (grammar_runtime.as_ref(), token_bytes_ref) {
                        super::grammar::mask::mask_invalid_tokens(rt, tb, &mut logits);
                    }
                    let (tok, lp_opt) = if want_logprobs {
                        let (t, lp) = sampler_pure::sample_token_with_logprob(
                            &mut logits,
                            sp,
                            &generated_tokens,
                        );
                        (t, Some(lp))
                    } else {
                        (
                            sampler_pure::sample_token(&mut logits, sp, &generated_tokens),
                            None,
                        )
                    };
                    if let (Some(acc), Some(lp_val)) = (logprobs_acc.as_mut(), lp_opt) {
                        acc.push(lp_val);
                    }
                    if let (Some(rt), Some(tb)) = (grammar_runtime.as_mut(), token_bytes_ref) {
                        let bytes = tb.get(tok as usize).map(|v| v.as_slice()).unwrap_or(&[]);
                        if !bytes.is_empty() {
                            rt.accept_bytes(bytes);
                        }
                    }
                    tok
                } else {
                    greedy_token
                };

                if loaded.eos_token_ids.contains(&next_token) {
                    finish_reason = "stop";
                    break;
                }
                generated_tokens.push(next_token);
                let fragment = loaded
                    .tokenizer
                    .decode(&[next_token], false)
                    .unwrap_or_default();
                decoded_text.push_str(&fragment);
                if let Some(sp) = splitter.as_mut() {
                    let _ = sp.feed(&fragment);
                    if sp.in_reasoning() {
                        reasoning_token_count += 1;
                    }
                }
                if let Some(tcs) = tc_splitter_ns.as_mut() {
                    let events = tcs.feed(&fragment);
                    if let Some(rt) = grammar_runtime.as_mut() {
                        if events
                            .iter()
                            .any(|e| matches!(e, super::registry::ToolCallEvent::ToolCallOpen))
                        {
                            rt.trigger();
                        }
                    }
                }
                if hit_stop_string(&decoded_text, &params.stop_strings) {
                    finish_reason = "stop";
                    strip_trailing_stop(&mut decoded_text, &params.stop_strings);
                    break;
                }
                // Grammar-dead termination (mirror of generate_once at
                // engine.rs:7856-7864).  Pop the offending token + re-
                // decode the surviving prefix.
                if grammar_runtime.as_ref().is_some_and(|rt| rt.is_dead()) {
                    finish_reason = "stop";
                    generated_tokens.pop();
                    decoded_text = loaded
                        .tokenizer
                        .decode(&generated_tokens, false)
                        .unwrap_or_default();
                    break;
                }
            }
        }
        let decode_duration = decode_started.elapsed();

        // Reasoning-text split at end-of-decode.  Mirror of generate_once
        // at engine.rs:7876-7879.
        let (content, reasoning_text) = match registration {
            Some(reg) if reg.has_reasoning() => super::registry::split_full_output_forced(
                reg,
                &decoded_text,
                params.reasoning_forced_open,
            ),
            _ => (decoded_text, None),
        };

        Ok(GenerationResult {
            text: content,
            reasoning_text,
            prompt_tokens: prompt_tokens.len(),
            completion_tokens: generated_tokens.len(),
            reasoning_tokens: if reasoning_enabled && reasoning_token_count > 0 {
                Some(reasoning_token_count)
            } else {
                None
            },
            finish_reason,
            prefill_duration,
            decode_duration,
            cached_tokens: 0, // iter-LCP scope.
            logprobs: logprobs_acc,
        })
    })();

    // Per-slot reset at exit — leave the slot clean for the next
    // request regardless of whether the kernel-forward sub-deferral
    // returned Ok or Err.  Belt-and-suspenders w/ the entry reset:
    // the iter-{2,3,4,5} ports will preserve this discipline so the
    // exit reset fires on every code path (success, EOS, error,
    // cancellation).  Mirror of Qwen35 iter-1 exit-reset.
    //
    // We swallow any error from the exit reset itself (logged via
    // tracing) so the kernel-forward result is the surface reported
    // to the caller — a failure to reset a slot at exit is a follow-up
    // observability concern, not a request failure (and is impossible
    // when the entry reset succeeded since bounds + buffer shape are
    // identical).
    for (layer_idx, buf) in multi_seq_kv.iter_mut().enumerate() {
        if let Err(e) = buf.reset_for_slot(slot_id) {
            tracing::warn!(
                "generate_gemma4_once_slot_aware: reset_for_slot at exit L{} \
                 failed (slot_id={}): {} — slot WILL be reset before next \
                 admission via the entry reset of the next call",
                layer_idx,
                slot_id.0,
                e
            );
        }
    }
    // ADR-040 iter-B4c-kernel iter-2B (2026-05-30) — exit reset on the
    // hybrid scaffold sibling.  Same swallow-on-error discipline as the
    // HB scaffold above — entry reset on the NEXT request will fire
    // if this fails (impossible at runtime since bounds + buffer shape
    // are identical to the entry-reset that already succeeded).
    if let Some(ref mut hybrid_scaffold) = multi_seq_kv_hybrid {
        for (layer_idx, buf) in hybrid_scaffold.iter_mut().enumerate() {
            if let Err(e) = buf.reset_for_slot(slot_id) {
                tracing::warn!(
                    "generate_gemma4_once_slot_aware: reset_for_slot (hybrid) at exit L{} \
                     failed (slot_id={}): {} — slot WILL be reset before next \
                     admission via the entry reset of the next call",
                    layer_idx,
                    slot_id.0,
                    e
                );
            }
        }
    }
    kernel_forward_result
}

/// **ADR-040 iter-B4c-kernel iter-3 (2026-05-30)** — slot-aware
/// Gemma 4 streaming chat generation against the persistent multi-seq
/// per-layer
/// [`crate::inference::models::gemma4::kv_cache::MultiSeqHbKvBuffers`]
/// scaffold (`GemmaLoadedModel.multi_seq_kv`) + the production-default
/// hybrid F16-K + TQ-HB-V sibling scaffold (`GemmaLoadedModel.
/// multi_seq_kv_hybrid`) instead of the legacy per-request inline
/// alloc.
///
/// **Direct mirror of `generate_gemma4_once_slot_aware`** (iter-1 +
/// iter-2A + iter-2B Generate-arm lift, per §6.1.31 + §6.1.32 +
/// §6.1.34) for the [`super::engine::Request::GenerateStream`] worker
/// arm.  iter-1+2A+2B landed the non-streaming Generate-arm lift +
/// kernel-forward step; iter-3 lands the streaming-arm lift onto the
/// same persistent scaffolds + per-slot reset + slot-aware
/// `forward_prefill_with_soft_tokens_slot_aware` kernel call.
///
/// Cross-architecture mirror of Qwen35 iter-C2d-cont-kernel iter-2
/// `engine_qwen35::generate_stream_qwen35_once_extended_slot_aware`
/// per §6.1.28 — same dispatch fork shape (`slot_id != SlotId(0)`
/// predicate at the worker arm), same take-and-restore borrow pattern,
/// same `reset_for_slot` entry+exit discipline, same SSE typed-error
/// emission via the events channel.
///
/// # Structural parallels with iter-1 (Generate-arm scaffold) +
/// iter-2A+2B (kernel-forward step)
///
/// 1. Bounds-checks `slot_id` against `multi_seq_kv[0].n_seqs` (bounds-
///    first per A2b §6.1.23 iter-1.5 cfa-finding-F5 ordering); surfaces
///    typed `capability_unsupported:` error event via SSE if slot OOR.
/// 2. Calls `MultiSeqHbKvBuffers::reset_for_slot(slot_id)` at entry on
///    every per-layer buffer — zeros the per-seq cursor for `slot_id`
///    only (other slots untouched).
/// 3. Mirrors entry-reset on the production-default hybrid scaffold
///    sibling (`multi_seq_kv_hybrid`) when `Some(_)` (HF2Q_HYBRID_KV=1
///    per H10 §6.1.11 default).
/// 4. Calls `loaded.weights.forward_prefill_with_soft_tokens_slot_aware(..)`
///    (the iter-2A landing per §6.1.32 + iter-2B routing per §6.1.34)
///    threading `slot_id` + both scaffolds + `&[]` soft_tokens (vision
///    streaming is iter-B4c-kernel-iter-5 scope; if `soft_tokens` is
///    non-empty the call surfaces a typed error event citing iter-5).
/// 5. The multi-token decode-loop body wrapping `forward_decode` calls
///    at `slot_id` is **iter-B4c-kernel-iter-2-decode scope** (the same
///    sub-deferral the Generate-arm lift surfaces at line 7955-7965).
///    iter-3 emits a typed `capability_unsupported:` error event citing
///    iter-2-decode + Done event with `finish_reason = "error"` after
///    a successful prefill, mirroring the Generate-arm IIFE pattern but
///    for the SSE surface.
/// 6. Calls `reset_for_slot(slot_id)` at exit on BOTH scaffolds — belt-
///    and-suspenders with the entry reset (mirror of iter-1's
///    discipline).
///
/// **Per-slot byte-equivalence at SlotId(0)** (H104 pin):
/// the `handle.slot_id != SlotId(0)` predicate at the worker arm short-
/// circuits AT the worker arm — `generate_stream_gemma4_once_slot_aware`
/// is NEVER called for SlotId(0).  Both SerialFifo (always SlotId(0))
/// and SlotAware + SlotId(0) route through the existing
/// `generate_stream_once` dispatch verbatim, preserving the H1/H2/H23/
/// H41/H44 byte-equivalence chain that A5*/C2a/C2b/C2c/B4c closed.
///
/// # Vision-augmented streaming deferral (iter-3 scope discipline)
///
/// When **any** of `soft_tokens` is non-empty, iter-3 emits a typed
/// `capability_unsupported:` error event citing iter-B4c-kernel-iter-5
/// and aborts.  Gemma 4's `Request::GenerateStream` channel does NOT
/// carry `deepstack` / `positions_flat` (Qwen3-VL-specific surfaces);
/// the streaming arm at `worker_run` already ignores those — see
/// engine.rs:5356-5360.  The text-only path (the majority case) gets
/// the full slot-aware throughput benefit at SlotId(N>0).
///
/// # Co-changes (iter-3 deliberately minimal — exact mirror of iter-2B
/// Generate-arm shape)
///
/// - Per-slot LCP / mid-prefill checkpoint storage is NOT applicable
///   to Gemma 4's streaming path today (the legacy `generate_stream_once`
///   uses `prompt_cache.store_with_fragments` after the decode loop —
///   tied to per-request `max_seq_len`).  Slot-aware LCP for Gemma 4
///   is pinned as **iter-B4c-kernel-iter-LCP** (parallel to Qwen35's
///   iter-C2d-cont-kernel-iter-LCP per §6.1.28).
/// - Vision-augmented streaming (soft_tokens any non-empty) returns a
///   typed error event citing **iter-B4c-kernel-iter-5** as the
///   implementing iter.
/// - Multi-token decode-loop body wrapping `forward_decode` at
///   `slot_id` is **iter-B4c-kernel-iter-2-decode scope** (mirror of
///   §6.1.32 iter-2A's same sub-deferral on the Generate arm).
///
/// # SSE event ordering (H109 pin)
///
/// The slot-aware streaming function emits SSE events in the same
/// order as `generate_stream_once`: per-token `Delta` events through
/// the splitter chain (none in iter-3 today because the decode loop
/// is sub-deferred — see iter-2-decode), followed by a terminal
/// `Done` event (or `Error` event on failure).  iter-3 currently
/// emits exactly:
/// - typed `Error` events on bounds + invariant failures + the
///   iter-2-decode sub-deferral (zero Delta events because the decode
///   loop is deferred);
/// - no terminal `Done` event when an Error event is emitted (the SSE
///   handler treats a terminal Error as a clean stream termination).
///
/// When iter-B4c-kernel-iter-2-decode lands, this fn's IIFE will grow
/// the per-token Delta emission loop + terminal Done — matching the
/// shape of `generate_stream_once` at engine.rs:9493-9486.
#[allow(clippy::too_many_arguments)]
fn generate_stream_gemma4_once_slot_aware(
    loaded: &mut GemmaLoadedModel,
    prompt_tokens: &[u32],
    soft_tokens: &[SoftTokenInjection<'_>],
    params: &SamplingParams,
    events: &tokio::sync::mpsc::Sender<super::sse::GenerationEvent>,
    // ADR-040 iter-B4c-kernel iter-2-decode-C (2026-05-30) — lifted from
    // `_registration` to `registration`.  iter-2-decode-A's greedy fast-
    // path body never engaged the reasoning splitter / tool-call
    // splitter; iter-2-decode-C wires `ReasoningSplitter` so streaming
    // reasoning-mode requests at SlotId(N>0) route Delta events to the
    // correct slot (Content vs Reasoning), and surfaces a typed
    // `iter-2-decode-C-stream-tool-call` defer when a ToolCallSplitter
    // would engage (Wave 3 W-B3 ToolCallStreamEmitter is ~200 LOC of
    // stateful incremental JSON parsing — out of scope for iter-2-decode-C).
    registration: Option<&super::registry::ModelRegistration>,
    cancellation_counter: Option<&std::sync::atomic::AtomicU64>,
    multi_seq_kv: &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqHbKvBuffers>,
    // ADR-040 iter-B4c-kernel iter-3 (2026-05-30) — production-default
    // hybrid F16-K + TQ-HB-V scaffold sibling param.  `Option<>` because
    // iter-C2c-cont provisions this field IFF the hybrid env-gate is
    // ON (DEFAULT per H10 §6.1.11).  Threaded verbatim through to the
    // model fn — same shape as the Generate-arm sibling per iter-2B.
    mut multi_seq_kv_hybrid: Option<
        &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqHybridKvBuffers>,
    >,
    // ADR-040 iter-B4c-kernel iter-2D / iter-2-decode-D (§6.1.46) —
    // dense F32 + legacy 4-bit scaffold siblings.
    mut multi_seq_kv_dense: Option<
        &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqDenseKvBuffers>,
    >,
    mut multi_seq_kv_mlx: Option<
        &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqMlxKvCache>,
    >,
    slot_id: SlotId,
) {
    // SSE emit helper: bumps cancellation counter + early-returns on
    // client disconnect.  Mirror of `generate_stream_once` shape +
    // Qwen35 `generate_stream_qwen35_once_extended_slot_aware` shape.
    macro_rules! send {
        ($ev:expr) => {
            if events.blocking_send($ev).is_err() {
                tracing::info!("SSE stream dropped by client; aborting gemma4 slot-aware decode");
                if let Some(c) = cancellation_counter {
                    c.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
                }
                return;
            }
        };
    }

    if prompt_tokens.is_empty() {
        send!(super::sse::GenerationEvent::Error(
            "generate_stream_gemma4_once_slot_aware: empty prompt_tokens".into()
        ));
        return;
    }
    if multi_seq_kv.is_empty() {
        send!(super::sse::GenerationEvent::Error(format!(
            "capability_unsupported: ADR-040 iter-B4c-kernel iter-3 — \
             multi_seq_kv is empty (C2c spawn-arm invariant: \
             provision_multi_seq_kv_for_slot_aware must produce one entry \
             per layer; ADR-040 §6.1.21). Operator: check spawn_with_mode \
             wiring in src/serve/api/engine.rs.",
        )));
        return;
    }
    // Bounds-first per A2b §6.1.23 iter-1.5 cfa-finding-F5 ordering.
    // Use the first layer's n_seqs as the canonical bound — A3a
    // construction guarantees every per-layer entry has the same
    // n_seqs (provisioned with max_slots).
    let n_seqs = multi_seq_kv[0].n_seqs;
    if slot_id.0 >= n_seqs {
        send!(super::sse::GenerationEvent::Error(format!(
            "capability_unsupported: ADR-040 iter-B4c-kernel iter-3 — \
             SlotOutOfRange slot={} max_slots={} (generate_stream_gemma4_\
             once_slot_aware)",
            slot_id.0, n_seqs,
        )));
        return;
    }

    // ADR-040 iter-B4c-kernel iter-5 (2026-05-30) — vision-augmented
    // streaming at SlotId(N>0) LIFTED.  iter-3 (§6.1.35) surfaced a
    // typed Error event citing iter-5 when `soft_tokens` was non-empty;
    // iter-5 REMOVES that abort path — the soft_tokens slice is now
    // threaded verbatim through to `forward_prefill_with_soft_tokens_slot_aware`
    // at the prefill call site below.  The slot-aware prefill kernel
    // (iter-2A landing per §6.1.32 + iter-2B routing per §6.1.34)
    // already accepts a `soft_tokens: &[SoftTokenInjection<'_>]`
    // parameter; iter-3's empty-only restriction was a scope-narrowing,
    // not a kernel limit.
    //
    // Mirror of Qwen35 iter-C2d-cont-kernel iter-4 §6.1.30's lift of
    // the `has_extension == true` branch in
    // `generate_stream_qwen35_once_extended_slot_aware`.  Same pattern:
    // the typed-error abort is REPLACED with the real kernel call +
    // t_post-advanced decode positioning (Gemma 4 path uses text-only
    // positions so the t_post logic reduces to the iter-3 form).
    //
    // Vision-augmented streaming at SlotId(N>0) now works end-to-end
    // through the kernel; the multi-token streaming decode-loop body
    // wrapping remains iter-2-decode scope (matches the text-only
    // streaming arm's surviving sub-deferral).
    let _ = soft_tokens; // Soft tokens are threaded through to the
                         // prefill call site below — this binding
                         // exists only to document iter-5's lift in
                         // the source-grep window.

    // Per-slot reset at entry — mirror of iter-1 Generate-arm entry-
    // reset discipline (engine.rs:7869-7874) for the streaming surface.
    // `reset_for_slot` zeros the per-seq cursor for `slot_id` on EVERY
    // per-layer entry; K/V packed + norms bytes are cursor-masked.
    for (layer_idx, buf) in multi_seq_kv.iter_mut().enumerate() {
        if let Err(e) = buf.reset_for_slot(slot_id) {
            send!(super::sse::GenerationEvent::Error(format!(
                "ADR-040 iter-B4c-kernel iter-3: reset_for_slot at entry \
                 L{layer_idx}: {e}",
            )));
            return;
        }
    }
    // ADR-040 iter-B4c-kernel iter-3 — entry reset on the hybrid
    // scaffold sibling.  Mirrors the HB scaffold entry-reset discipline
    // above for the production-default regime.  Uses `.as_deref_mut()`
    // to reborrow so the model fn call below can re-take the same
    // `Option<&mut Vec<_>>` shape.
    if let Some(ref mut hybrid_scaffold) = multi_seq_kv_hybrid {
        for (layer_idx, buf) in hybrid_scaffold.iter_mut().enumerate() {
            if let Err(e) = buf.reset_for_slot(slot_id) {
                send!(super::sse::GenerationEvent::Error(format!(
                    "ADR-040 iter-B4c-kernel iter-3: reset_for_slot at \
                     entry (hybrid) L{layer_idx}: {e}",
                )));
                return;
            }
        }
    }

    // ADR-040 iter-B4c-kernel iter-3 — kernel-forward call mirroring
    // iter-2A/2B Generate-arm shape (engine.rs:7920-7966).  Calls
    // `forward_prefill_with_soft_tokens_slot_aware` (the iter-2A landing
    // per §6.1.32 + iter-2B routing per §6.1.34).  Returns the first
    // decode token; the multi-token decode-loop body wrapping is
    // iter-B4c-kernel-iter-2-decode scope (same sub-deferral the
    // Generate-arm lift surfaces).
    //
    // ADR-040 iter-B4c-kernel iter-5 (2026-05-30) — `soft_tokens` is
    // now threaded VERBATIM through to the slot-aware prefill kernel
    // (was `&[]` pre-iter-5; iter-5 lifted the vision-augmented
    // streaming branch's abort above).  Mirror of Qwen35 iter-4 §6.1.30
    // streaming has_extension lift.
    let max_decode_tokens = params.max_tokens.max(1);
    let prefill_result: Result<u32> = loaded.weights.forward_prefill_with_soft_tokens_slot_aware(
        prompt_tokens,
        soft_tokens, // GenerateStream-arm: iter-5 (§6.1.37) lifts
        // the vision-augmented streaming branch —
        // soft-token overrides are threaded through
        // to the slot-aware prefill kernel verbatim.
        max_decode_tokens,
        &mut loaded.ctx,
        slot_id,
        multi_seq_kv,
        multi_seq_kv_hybrid.as_deref_mut(),
        // ADR-040 iter-2D + iter-2C (§6.1.46): dense F32 + legacy
        // 4-bit scaffold siblings (None when env-gate is off).
        multi_seq_kv_dense.as_deref_mut(),
        multi_seq_kv_mlx.as_deref_mut(),
    );

    // ADR-040 iter-B4c-kernel iter-2-decode-A (2026-05-30) — GenerateStream-arm
    // decode-loop body landed (greedy fast-path).
    //
    // ADR-040 iter-B4c-kernel iter-2-decode-C (2026-05-30) — FULL streaming
    // sampler/grammar/stop-strings/logprobs/reasoning-text surface lands.
    // Mirror of `generate_stream_once` at engine.rs:11008+ stripped to
    // the load-bearing decode loop (no PromptCache replay, no LCP
    // probe — those are SerialFifo-only optimizations not engaged at
    // SlotId(N>0) per the iter-LCP scope).
    //
    // ADR-040 iter-B4c-kernel iter-2-decode-C-stream-tool-call SHIPPED
    // 2026-05-30 (§6.1.48) — the iter-2-decode-C surviving sub-deferral
    // (streaming tool-call body emission via Wave 3 W-B3
    // `ToolCallStreamEmitter`) is LIFTED.  The slot-aware streaming
    // body now threads the same `tool_splitter` + `tool_call_body`
    // accumulator + `tool_call_emitter` Option + `tool_call_index` +
    // `saw_tool_call` per-stream state as `generate_stream_once` at
    // engine.rs:12140-12317.  When a `ToolCallSplitter` is registered
    // for the model AND a tool-call body grammar is requested
    // (`grammar_kind ∈ {ToolCallBodyAuto, ToolCallBodyRequired}`), the
    // streaming arm now drives `ToolCallStreamEmitter::advance` per
    // ToolCallText fragment + `finalize` per ToolCallClose, matching
    // the non-slot-aware shape verbatim.  The iter-2-decode-C typed-
    // error surface (`stream_tool_call_engaged` short-circuit at the
    // `Ok(first_decode_token)` arm of `match prefill_result`) is
    // REMOVED — every prefill-Ok branch now runs the unified tool-
    // call-aware decode loop.  The `iter-2-decode-C-stream-tool-call
    // per ADR-040 §6.1.39` label substring is preserved as a doc-
    // comment cite for H87 forward-pointer discoverability (operator-
    // grep'able), but the typed `MultiSeqError::CapabilityUnsupported
    // { capability: "...stream-tool-call..." }` constructor + the
    // associated SSE Error event are GONE — replaced by the real
    // incremental tool-call argument streaming path.
    //
    // iter-2-decode-C-stream-tool-call per ADR-040 §6.1.39 — sub-deferral
    // CLOSED at §6.1.48 (this iter).  Substring preserved as a doc-
    // comment cite so `grep "iter-2-decode-C-stream-tool-call per"`
    // still discovers the historical scope-narrowing decision in the
    // source tree.
    match prefill_result {
        Ok(first_decode_token) => {
            // iter-2-decode-C-stream-tool-call per ADR-040 §6.1.39 —
            // historical doc-cite for H87 forward-pointer discoverability.
            // The typed-error `MultiSeqError::CapabilityUnsupported`
            // surface that pre-iter-2-decode-C-stream-tool-call landed
            // at this branch entry is REMOVED; the unified body below
            // runs the real Wave 3 W-B3 `ToolCallStreamEmitter` path.

            // ── Sampler / grammar / logprobs config (mirror of
            // generate_stream_once at engine.rs:11453+).
            let sample_logits = params.temperature > 0.0
                || params.top_k > 0
                || params.top_p < 1.0
                || params.repetition_penalty != 1.0
                || !params.logit_bias.is_empty()
                || params.grammar.is_some()
                || params.logprobs;
            let sampler_params = if sample_logits {
                Some(SamplerParams {
                    temperature: params.temperature as f64,
                    top_p: params.top_p as f64,
                    top_k: params.top_k,
                    min_p: 0.0,
                    repetition_penalty: effective_repetition_penalty(params),
                    max_tokens: params.max_tokens,
                })
            } else {
                None
            };

            let mut grammar_runtime: Option<super::grammar::GrammarRuntime> =
                match params.grammar.as_ref() {
                    Some(g) => {
                        let start_rule_id = match g.rule_id("root") {
                            Some(id) => id,
                            None => {
                                send!(super::sse::GenerationEvent::Error(
                                    "grammar has no root rule".into()
                                ));
                                return;
                            }
                        };
                        let mut rt =
                            match super::grammar::GrammarRuntime::new(g.clone(), start_rule_id) {
                                Some(r) => r,
                                None => {
                                    send!(super::sse::GenerationEvent::Error(
                                        "grammar runtime init failed".into()
                                    ));
                                    return;
                                }
                            };
                        if matches!(params.grammar_kind, GrammarKind::ToolCallBodyAuto) {
                            rt.set_awaiting_trigger(true);
                        }
                        Some(rt)
                    }
                    None => None,
                };
            let token_bytes_ref: Option<&[Vec<u8>]> = params.token_bytes.as_deref().map(|v| &v[..]);

            // Reasoning splitter classifies each fragment into
            // Content vs Reasoning DeltaKind.  When `None` (model
            // has no reasoning markers registered), every fragment
            // routes to Content.
            let mut reason_splitter = registration.and_then(|r| {
                super::registry::make_reasoning_splitter(r, params.reasoning_forced_open)
            });

            // ADR-040 iter-2-decode-C-stream-tool-call per §6.1.48 —
            // tool-call splitter classifies the post-reasoning
            // Content stream into in/out-of-tool-call spans
            // (mirror of generate_stream_once at engine.rs:12140-
            // 12152).  Composition: reasoning splitter first; any
            // Content-classified fragment then flows into the
            // tool-call splitter via `route_content`.  When the
            // model has no tool-call markers registered, the
            // splitter is `None` and every fragment routes
            // verbatim through `Delta { kind: Content, .. }`
            // (byte-equivalent to the pre-iter-2-decode-C-stream-
            // tool-call shape).
            let mut tool_splitter =
                registration.and_then(super::registry::ToolCallSplitter::from_registration);
            let mut tool_call_body: String = String::new();
            let mut tool_call_index: usize = 0;
            let mut saw_tool_call: bool = false;
            let mut tool_call_emitter: Option<ToolCallStreamEmitter> = None;
            let tool_call_policy = params.tool_call_policy;

            // ADR-040 iter-2-decode-C-stream-tool-call per §6.1.48 —
            // EventSink wrapper for ToolCallStreamEmitter::{advance,
            // finalize}, which take `&EventSink<'_>` instead of the
            // raw `&Sender`.  The slot-aware fn has no streaming-
            // origin capture (no PromptCache store on slot-aware
            // path — that's iter-LCP scope per §6.1.39), so we use
            // the passive `EventSink::new` constructor.  The
            // wrapper forwards every blocking_send call verbatim
            // to the underlying sender.
            let event_sink = EventSink::new(events);

            let want_logprobs = params.logprobs;
            let want_log_per_token = want_logprobs;

            // Closure: classify a fragment + emit Delta events
            // through the reasoning splitter → tool-call splitter
            // pipeline.  Returns Err if SSE send failed (signals
            // stream cancellation).
            //
            // We can't return early from a closure to the outer fn,
            // so the closure produces `Result<(), ()>` and the
            // caller checks + breaks the loop.
            //
            // ADR-040 iter-2-decode-C-stream-tool-call per §6.1.48
            // — closure signature widened from the iter-2-decode-C
            // shape `(events, splitter, fragment)` to
            // `(events_sink, splitter, tool_splitter, body,
            // tc_index, saw_tc, emitter, grammar_runtime,
            // fragment, reg)` to thread the per-call tool-call
            // streaming state through.  Mirror of
            // `generate_stream_once::emit_fragment` at engine.rs:
            // 12323-12382 + `route_content` at 12210-12317.
            let emit_fragment = |event_sink: &EventSink<'_>,
                                 splitter: &mut Option<super::registry::ReasoningSplitter>,
                                 tool_splitter: &mut Option<super::registry::ToolCallSplitter>,
                                 body: &mut String,
                                 tc_index: &mut usize,
                                 saw_tc: &mut bool,
                                 emitter: &mut Option<ToolCallStreamEmitter>,
                                 grammar_runtime: &mut Option<super::grammar::GrammarRuntime>,
                                 fragment: &str,
                                 reg: Option<&super::registry::ModelRegistration>|
             -> Result<(), ()> {
                // Inner helper: route a Content-classified text
                // run through the ToolCallSplitter (when
                // present) or emit as a Content Delta event
                // verbatim.  Mirror of
                // generate_stream_once::route_content shape.
                let route_content =
                    |tool_splitter: &mut Option<super::registry::ToolCallSplitter>,
                     body: &mut String,
                     tc_index: &mut usize,
                     saw_tc: &mut bool,
                     emitter: &mut Option<ToolCallStreamEmitter>,
                     grammar_runtime: &mut Option<super::grammar::GrammarRuntime>,
                     text: &str,
                     reg: Option<&super::registry::ModelRegistration>|
                     -> Result<(), ()> {
                        if text.is_empty() {
                            return Ok(());
                        }
                        let Some(tcs) = tool_splitter.as_mut() else {
                            // No tool markers registered — original behavior.
                            if event_sink
                                .blocking_send(super::sse::GenerationEvent::Delta {
                                    kind: super::sse::DeltaKind::Content,
                                    text: text.to_string(),
                                })
                                .is_err()
                            {
                                return Err(());
                            }
                            return Ok(());
                        };
                        for ev in tcs.feed(text) {
                            match ev {
                                super::registry::ToolCallEvent::Content(t) => {
                                    if !t.is_empty()
                                        && event_sink
                                            .blocking_send(super::sse::GenerationEvent::Delta {
                                                kind: super::sse::DeltaKind::Content,
                                                text: t,
                                            })
                                            .is_err()
                                    {
                                        return Err(());
                                    }
                                }
                                super::registry::ToolCallEvent::ToolCallOpen => {
                                    body.clear();
                                    // Wave 3 W-B3 incremental:
                                    // fresh emitter for THIS call.
                                    *emitter = Some(ToolCallStreamEmitter::new(
                                        reg.map(|r| r.family),
                                        *tc_index,
                                    ));
                                    // Wave 2.6 W-α5 Q2: arm grammar trigger.
                                    if let Some(rt) = grammar_runtime.as_mut() {
                                        rt.trigger();
                                    }
                                }
                                super::registry::ToolCallEvent::ToolCallText(t) => {
                                    body.push_str(&t);
                                    if let Some(em) = emitter.as_mut() {
                                        em.advance(body, event_sink)?;
                                    }
                                }
                                super::registry::ToolCallEvent::ToolCallClose => {
                                    let body_dump = std::mem::take(body);
                                    let mut em = emitter.take().unwrap_or_else(|| {
                                        ToolCallStreamEmitter::new(reg.map(|r| r.family), *tc_index)
                                    });
                                    em.finalize(
                                        body_dump,
                                        reg,
                                        tool_call_policy,
                                        tc_index,
                                        saw_tc,
                                        event_sink,
                                    )?;
                                }
                            }
                        }
                        Ok(())
                    };

                if fragment.is_empty() {
                    return Ok(());
                }
                if let Some(sp) = splitter.as_mut() {
                    for (slot, frag) in sp.feed(fragment) {
                        match slot {
                            super::registry::SplitSlot::Reasoning => {
                                if !frag.is_empty()
                                    && event_sink
                                        .blocking_send(super::sse::GenerationEvent::Delta {
                                            kind: super::sse::DeltaKind::Reasoning,
                                            text: frag,
                                        })
                                        .is_err()
                                {
                                    return Err(());
                                }
                            }
                            super::registry::SplitSlot::Content => {
                                route_content(
                                    tool_splitter,
                                    body,
                                    tc_index,
                                    saw_tc,
                                    emitter,
                                    grammar_runtime,
                                    &frag,
                                    reg,
                                )?;
                            }
                        }
                    }
                } else {
                    // No reasoning splitter — route everything as Content.
                    route_content(
                        tool_splitter,
                        body,
                        tc_index,
                        saw_tc,
                        emitter,
                        grammar_runtime,
                        fragment,
                        reg,
                    )?;
                }
                Ok(())
            };

            // First decode token (sampler path re-derives from live
            // logits; greedy path reuses prefill argmax).
            let mut next_token = if sample_logits {
                let sp = sampler_params.as_ref().expect("sample_logits gate");
                let mut logits: Vec<f32> = match loaded.weights.logits_view() {
                    Ok(s) => s.to_vec(),
                    Err(e) => {
                        send!(super::sse::GenerationEvent::Error(format!(
                            "gemma4 stream slot-aware logits_view failed: {e:#}",
                        )));
                        return;
                    }
                };
                if !params.logit_bias.is_empty() {
                    let v = logits.len();
                    for (&id, &bias) in &params.logit_bias {
                        let idx = id as usize;
                        if idx < v {
                            logits[idx] += bias;
                        }
                    }
                }
                if let (Some(rt), Some(tb)) = (grammar_runtime.as_ref(), token_bytes_ref) {
                    super::grammar::mask::mask_invalid_tokens(rt, tb, &mut logits);
                }
                let (tok, lp_opt) = if want_logprobs {
                    let (t, lp) = sampler_pure::sample_token_with_logprob(&mut logits, sp, &[]);
                    (t, Some(lp))
                } else {
                    (sampler_pure::sample_token(&mut logits, sp, &[]), None)
                };
                if let (Some(_lp), true) = (lp_opt, want_log_per_token) {
                    // Per-token logprob streaming uses the
                    // streaming Logprobs event; for the iter-2-
                    // decode-C scope we emit a minimal
                    // single-entry chunk.  The SSE encoder at
                    // sse.rs:303 handles the rest.
                    // Implementation: we emit the raw chosen-token
                    // logprob as a degenerate ChoiceLogprobs
                    // (single entry) — fuller top-K shape is
                    // iter-LCP/iter-G scope, not iter-2-decode-C.
                    // For now we skip the per-token Logprobs event
                    // and let the final Done event carry the
                    // aggregate; full per-token streaming is the
                    // generate_stream_once shape which uses
                    // ToolCallSplitter — that's iter-2-decode-C-
                    // stream-tool-call scope.
                    // NOTE: this is intentionally a degraded surface
                    // — the request still completes correctly; the
                    // per-token logprob granularity for streaming
                    // is the documented sub-deferral.
                }
                if let (Some(rt), Some(tb)) = (grammar_runtime.as_mut(), token_bytes_ref) {
                    let bytes = tb.get(tok as usize).map(|v| v.as_slice()).unwrap_or(&[]);
                    if !bytes.is_empty() {
                        rt.accept_bytes(bytes);
                    }
                }
                tok
            } else {
                first_decode_token
            };

            let mut generated_tokens: Vec<u32> = Vec::with_capacity(max_decode_tokens);
            let mut completion_token_count: usize = 0;
            let mut finish_reason: &'static str = "length";
            let mut decoded_running = String::new();

            let first_fragment = loaded
                .tokenizer
                .decode(&[next_token], false)
                .unwrap_or_default();
            decoded_running.push_str(&first_fragment);
            if emit_fragment(
                &event_sink,
                &mut reason_splitter,
                &mut tool_splitter,
                &mut tool_call_body,
                &mut tool_call_index,
                &mut saw_tool_call,
                &mut tool_call_emitter,
                &mut grammar_runtime,
                &first_fragment,
                registration,
            )
            .is_err()
            {
                tracing::info!("SSE stream dropped by client; aborting gemma4 slot-aware decode");
                if let Some(c) = cancellation_counter {
                    c.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
                }
                return;
            }
            completion_token_count += 1;

            let mut decode_err: Option<anyhow::Error> = None;
            if loaded.eos_token_ids.contains(&next_token) {
                finish_reason = "stop";
            } else if hit_stop_string(&decoded_running, &params.stop_strings) {
                finish_reason = "stop";
            } else {
                generated_tokens.push(next_token);
                for _ in 1..max_decode_tokens {
                    let pos = prompt_tokens.len() + generated_tokens.len() - 1;
                    let mut p: Option<crate::inference::models::gemma4::profile::TokenProfile> =
                        None;
                    let r = loaded.weights.forward_decode_slot_aware(
                        next_token,
                        pos,
                        &mut loaded.ctx,
                        &mut p,
                        slot_id,
                        multi_seq_kv,
                        multi_seq_kv_hybrid.as_deref_mut(),
                        // ADR-040 iter-2-decode-D (§6.1.46).
                        multi_seq_kv_dense.as_deref_mut(),
                        multi_seq_kv_mlx.as_deref_mut(),
                    );
                    let greedy_token = match r {
                        Ok(t) => t,
                        Err(e) => {
                            decode_err = Some(e);
                            break;
                        }
                    };

                    next_token = if sample_logits {
                        let sp = sampler_params.as_ref().expect("sample_logits gate");
                        let mut logits: Vec<f32> = match loaded.weights.logits_view() {
                            Ok(s) => s.to_vec(),
                            Err(e) => {
                                decode_err = Some(e);
                                break;
                            }
                        };
                        if !params.logit_bias.is_empty() {
                            let v = logits.len();
                            for (&id, &bias) in &params.logit_bias {
                                let idx = id as usize;
                                if idx < v {
                                    logits[idx] += bias;
                                }
                            }
                        }
                        if let (Some(rt), Some(tb)) = (grammar_runtime.as_ref(), token_bytes_ref) {
                            super::grammar::mask::mask_invalid_tokens(rt, tb, &mut logits);
                        }
                        let (tok, _lp_opt) = if want_logprobs {
                            let (t, lp) = sampler_pure::sample_token_with_logprob(
                                &mut logits,
                                sp,
                                &generated_tokens,
                            );
                            (t, Some(lp))
                        } else {
                            (
                                sampler_pure::sample_token(&mut logits, sp, &generated_tokens),
                                None,
                            )
                        };
                        if let (Some(rt), Some(tb)) = (grammar_runtime.as_mut(), token_bytes_ref) {
                            let bytes = tb.get(tok as usize).map(|v| v.as_slice()).unwrap_or(&[]);
                            if !bytes.is_empty() {
                                rt.accept_bytes(bytes);
                            }
                        }
                        tok
                    } else {
                        greedy_token
                    };

                    if loaded.eos_token_ids.contains(&next_token) {
                        finish_reason = "stop";
                        break;
                    }
                    generated_tokens.push(next_token);
                    let fragment = loaded
                        .tokenizer
                        .decode(&[next_token], false)
                        .unwrap_or_default();
                    decoded_running.push_str(&fragment);
                    if emit_fragment(
                        &event_sink,
                        &mut reason_splitter,
                        &mut tool_splitter,
                        &mut tool_call_body,
                        &mut tool_call_index,
                        &mut saw_tool_call,
                        &mut tool_call_emitter,
                        &mut grammar_runtime,
                        &fragment,
                        registration,
                    )
                    .is_err()
                    {
                        tracing::info!(
                            "SSE stream dropped by client; aborting gemma4 slot-aware decode"
                        );
                        if let Some(c) = cancellation_counter {
                            c.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
                        }
                        return;
                    }
                    completion_token_count += 1;

                    if hit_stop_string(&decoded_running, &params.stop_strings) {
                        finish_reason = "stop";
                        break;
                    }
                    if grammar_runtime.as_ref().is_some_and(|rt| rt.is_dead()) {
                        finish_reason = "stop";
                        break;
                    }
                }
            }
            // ADR-040 iter-2-decode-C-stream-tool-call per §6.1.48
            // — finish_reason override: per OpenAI tool-calls spec,
            // when ANY tool-call closed during the stream
            // (`saw_tool_call` latched true by ToolCallStreamEmitter
            // ::finalize / emit_streaming_tool_call_close), the
            // terminal finish_reason is `"tool_calls"` regardless
            // of whether the grammar exhausted (which would
            // otherwise read as `"stop"`) or the decode loop hit
            // max_tokens (`"length"`).  Mirror of
            // generate_stream_once at engine.rs:12754+ shape.
            if saw_tool_call {
                finish_reason = "tool_calls";
            }
            if let Some(e) = decode_err {
                send!(super::sse::GenerationEvent::Error(format!(
                    "gemma4 stream slot-aware decode failed: {e:#}",
                )));
            } else {
                send!(super::sse::GenerationEvent::Done {
                    finish_reason,
                    prompt_tokens: prompt_tokens.len(),
                    completion_tokens: completion_token_count,
                    stats: super::sse::StreamStats::default(),
                });
            }
        }
        Err(e) => {
            send!(super::sse::GenerationEvent::Error(format!(
                "gemma4 stream slot-aware prefill failed: {e:#}",
            )));
        }
    }

    // Per-slot reset at exit — leave the slot clean for the next
    // request regardless of whether the kernel-forward returned Ok or
    // Err.  Belt-and-suspenders with the entry reset; mirrors iter-1
    // Generate-arm exit-reset discipline.  Errors swallowed via
    // tracing::warn — a failure to reset a slot at exit is observability,
    // not request failure (entry reset on the next request will fire).
    for (layer_idx, buf) in multi_seq_kv.iter_mut().enumerate() {
        if let Err(e) = buf.reset_for_slot(slot_id) {
            tracing::warn!(
                "generate_stream_gemma4_once_slot_aware: reset_for_slot at \
                 exit L{} failed (slot_id={}): {} — slot WILL be reset \
                 before next admission via the entry reset of the next call",
                layer_idx,
                slot_id.0,
                e
            );
        }
    }
    // ADR-040 iter-B4c-kernel iter-3 — exit reset on the hybrid
    // scaffold sibling.  Same swallow-on-error discipline as the HB
    // scaffold above.
    if let Some(ref mut hybrid_scaffold) = multi_seq_kv_hybrid {
        for (layer_idx, buf) in hybrid_scaffold.iter_mut().enumerate() {
            if let Err(e) = buf.reset_for_slot(slot_id) {
                tracing::warn!(
                    "generate_stream_gemma4_once_slot_aware: reset_for_slot \
                     (hybrid) at exit L{} failed (slot_id={}): {} — slot \
                     WILL be reset before next admission via the entry \
                     reset of the next call",
                    layer_idx,
                    slot_id.0,
                    e
                );
            }
        }
    }
}

/// **ADR-040 iter-B4c-kernel iter-4 (2026-05-30)** — slot-aware
/// chat-as-embedder entry that routes the Gemma 4 worker hot path's
/// **Embed** dispatch through the persistent multi-seq per-layer
/// [`crate::inference::models::gemma4::kv_cache::MultiSeqHbKvBuffers`]
/// scaffold (`GemmaLoadedModel.multi_seq_kv`) + the production-default
/// hybrid F16-K + TQ-HB-V sibling scaffold (`GemmaLoadedModel.
/// multi_seq_kv_hybrid`) instead of a per-request fresh `forward_embed_last`
/// invocation against the legacy in-place `MlxModelWeights` cache.
///
/// **Direct mirror of `generate_gemma4_once_slot_aware`** (iter-1 +
/// iter-2A + iter-2B Generate-arm lift, per §6.1.31 + §6.1.32 + §6.1.34)
/// AND `generate_stream_gemma4_once_slot_aware` (iter-3 GenerateStream-
/// arm lift, per §6.1.35) for the [`super::engine::Request::Embed`]
/// worker arm.  iter-1+2A+2B landed the non-streaming Generate-arm lift
/// + kernel-forward step; iter-3 landed the streaming-arm lift; iter-4
/// lands the embed-arm lift onto the same persistent scaffolds + per-
/// slot reset + slot-aware `forward_prefill_with_soft_tokens_slot_aware`
/// kernel call.
///
/// **Cross-architecture mirror of Qwen35 iter-C2d-cont-kernel iter-3
/// `engine_qwen35::embed_qwen35_slot_aware` per §6.1.29** — same
/// dispatch fork shape (`slot_id != SlotId(0)` predicate at the worker
/// arm), same take-and-restore borrow pattern, same `reset_for_slot`
/// entry+exit discipline, same `Result<Vec<f32>>` return surface (L2-
/// normalized hidden vector).
///
/// # Why iter-4 is the smallest of the iter-{1,2A/2B,3,4,5} Gemma 4 ports
///
/// Embed runs **exactly one** `forward_prefill_with_soft_tokens_slot_aware`
/// call against the slot with `max_decode_tokens=0` (matching the
/// existing `MlxModelWeights::forward_embed_last` shape at
/// `forward_prefill.rs:2271-2332`), reads the L2-normalized hidden
/// vector out of `loaded.weights.activations.norm_out`, and exits.  No
/// decode loop (iter-2-decode sub-deferral does **NOT** apply), no SSE
/// channel, no soft-token injections (the `Request::Embed` variant
/// carries only `prompt_tokens`), no prompt-cache HIT fast-path (same
/// shape as non-slot-aware `forward_embed_last`).
///
/// # Structural parallels with iter-1+2A+2B (Generate) + iter-3 (Stream)
///
/// 1. Bounds-checks `slot_id` against `multi_seq_kv[0].n_seqs` (bounds-
///    first per A2b §6.1.23 iter-1.5 cfa-finding-F5 ordering); surfaces
///    typed `anyhow::Error` with `capability_unsupported:` prefix + cite.
/// 2. Calls `MultiSeqHbKvBuffers::reset_for_slot(slot_id)` at entry on
///    every per-layer buffer — zeros the per-seq cursor for `slot_id`
///    only (other slots untouched).
/// 3. Mirrors entry-reset on the production-default hybrid scaffold
///    sibling (`multi_seq_kv_hybrid`) when `Some(_)` (HF2Q_HYBRID_KV=1
///    per H10 §6.1.11 default).
/// 4. Calls `loaded.weights.forward_prefill_with_soft_tokens_slot_aware(..)`
///    (the iter-2A landing per §6.1.32 + iter-2B routing per §6.1.34)
///    with `&[]` soft_tokens + `max_decode_tokens=0`, threading
///    `slot_id` + both scaffolds.  The return value (first decode token)
///    is intentionally DISCARDED — embed reads the hidden state from
///    `loaded.weights.activations.norm_out`, not the next-token argmax.
/// 5. Reads the L2-normalized hidden vector out of
///    `loaded.weights.activations.norm_out` (length
///    `loaded.weights.hidden_size`) — byte-equivalent to the tail of
///    `MlxModelWeights::forward_embed_last` at forward_prefill.rs:2306-2331.
/// 6. Calls `reset_for_slot(slot_id)` at exit on BOTH scaffolds — belt-
///    and-suspenders with the entry reset (mirror of iter-1's exit
///    discipline). The embed path has NO intermediate paths that could
///    bypass the entry reset, but the exit reset preserves the cross-
///    iter exit-discipline pattern so the slot is always clean at
///    handoff for the next request to land at this slot.
///
/// **Per-slot byte-equivalence at SlotId(0)** (H110 pin):
/// the `handle.slot_id != SlotId(0)` predicate at the worker arm short-
/// circuits AT the worker arm — `embed_gemma4_slot_aware` is NEVER
/// called for SlotId(0).  Both SerialFifo (always SlotId(0)) and
/// SlotAware + SlotId(0) route through the existing
/// `g.weights.forward_embed_last(&prompt_tokens, &mut g.ctx)` dispatch
/// verbatim, preserving the H1/H2/H23/H41/H44/H77/H104 byte-equivalence
/// chain that A5*/C2a/C2b/C2c/B4c/iter-1/iter-3 closed.
///
/// # Multi-token decode-loop sub-deferral does NOT apply
///
/// Unlike iter-2A/2B (Generate) + iter-3 (GenerateStream) which both
/// emit a typed `iter-B4c-kernel-iter-2-decode` sub-deferral after the
/// prefill Ok branch, iter-4 does **NOT** emit that sub-deferral: embed
/// has no decode loop, so the multi-token decode-loop body wrapping is
/// structurally N/A.  The prefill's first-decode-token return is
/// discarded (it would be the prefill argmax, not part of the embed
/// output).  This is the load-bearing structural simplification that
/// makes iter-4 the smallest of the remaining Gemma 4 worker-arm ports.
///
/// # Errors
/// - `prompt_tokens.is_empty()` (matches the existing
///   `forward_embed_last` precondition).
/// - `slot_id.0 >= multi_seq_kv[0].n_seqs` (bounds-first; surfaces typed
///   `anyhow::Error` with `capability_unsupported:` prefix +
///   `iter-B4c-kernel iter-4` cite).
/// - `reset_for_slot` failure propagates with `iter-B4c-kernel iter-4`
///   context.
/// - `forward_prefill_with_soft_tokens_slot_aware` failure propagates
///   with the usual context.
/// - `norm_out` read failure propagates.
#[allow(clippy::too_many_arguments)]
fn embed_gemma4_slot_aware(
    loaded: &mut GemmaLoadedModel,
    prompt_tokens: &[u32],
    multi_seq_kv: &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqHbKvBuffers>,
    // ADR-040 iter-B4c-kernel iter-4 (2026-05-30) — production-default
    // hybrid F16-K + TQ-HB-V scaffold sibling param.  `Option<>` because
    // iter-C2c-cont provisions this field IFF the hybrid env-gate is
    // ON (DEFAULT per H10 §6.1.11).  Threaded verbatim through to the
    // model fn — same shape as the Generate-arm sibling per iter-2B
    // + the GenerateStream-arm sibling per iter-3.
    //
    // `mut` binding so the orchestrator body can do entry+exit
    // `reset_for_slot` via `if let Some(ref mut _) = multi_seq_kv_hybrid`
    // AND pass an `as_deref_mut()` reborrow to the model fn call below.
    mut multi_seq_kv_hybrid: Option<
        &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqHybridKvBuffers>,
    >,
    // ADR-040 iter-B4c-kernel iter-2D / iter-2C (§6.1.46) — dense F32 +
    // legacy 4-bit scaffold siblings.  Threaded as Option<&mut> per the
    // iter-3 + iter-5 precedent.  Embed-arm has NO decode loop (H180),
    // so these are consumed only by the iter-2D / iter-2C prefill
    // dispatch-fork branches in `forward_prefill_with_soft_tokens_slot_aware`.
    mut multi_seq_kv_dense: Option<
        &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqDenseKvBuffers>,
    >,
    mut multi_seq_kv_mlx: Option<
        &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqMlxKvCache>,
    >,
    slot_id: SlotId,
) -> Result<Vec<f32>> {
    anyhow::ensure!(
        !prompt_tokens.is_empty(),
        "embed_gemma4_slot_aware: empty prompt_tokens \
         (ADR-040 iter-B4c-kernel iter-4)"
    );
    anyhow::ensure!(
        !multi_seq_kv.is_empty(),
        "embed_gemma4_slot_aware: multi_seq_kv is empty \
         (C2c spawn-arm invariant: provision_multi_seq_kv_for_slot_aware \
          must produce one entry per layer; ADR-040 §6.1.21)"
    );
    // Bounds-first per A2b §6.1.23 iter-1.5 cfa-finding-F5 ordering.
    // Use the first layer's n_seqs as the canonical bound — A3a
    // construction guarantees every per-layer entry has the same
    // n_seqs (provisioned with max_slots).
    let n_seqs = multi_seq_kv[0].n_seqs;
    anyhow::ensure!(
        slot_id.0 < n_seqs,
        "embed_gemma4_slot_aware: SlotOutOfRange slot={} max_slots={} \
         (ADR-040 iter-B4c-kernel iter-4)",
        slot_id.0,
        n_seqs,
    );

    // Per-slot reset at entry — the persistent cache may carry stale
    // bytes from a prior request on this slot.  `reset_for_slot` zeros
    // the per-seq cursor for `slot_id` on EVERY per-layer entry; K/V
    // packed + norms bytes are cursor-masked (see layout proof at
    // `MultiSeqHbKvBuffers::reset_for_slot`).
    //
    // Mirror of iter-1 Generate-arm + iter-3 GenerateStream-arm entry-
    // reset discipline.
    for (layer_idx, buf) in multi_seq_kv.iter_mut().enumerate() {
        buf.reset_for_slot(slot_id).map_err(|e| {
            anyhow::anyhow!(
                "embed_gemma4_slot_aware: reset_for_slot at entry L{layer_idx}: {e} \
             (ADR-040 iter-B4c-kernel iter-4)"
            )
        })?;
    }
    // ADR-040 iter-B4c-kernel iter-4 — entry reset on the hybrid
    // scaffold sibling.  Mirrors the HB scaffold entry-reset discipline
    // above for the production-default regime.  Uses `.as_deref_mut()`
    // to reborrow so the model fn call below can re-take the same
    // `Option<&mut Vec<_>>` shape (the borrow ends with this for-loop
    // scope).
    if let Some(ref mut hybrid_scaffold) = multi_seq_kv_hybrid {
        for (layer_idx, buf) in hybrid_scaffold.iter_mut().enumerate() {
            buf.reset_for_slot(slot_id).map_err(|e| {
                anyhow::anyhow!(
                    "embed_gemma4_slot_aware: reset_for_slot at entry (hybrid) \
                 L{layer_idx}: {e} (ADR-040 iter-B4c-kernel iter-4)"
                )
            })?;
        }
    }

    // ADR-040 iter-B4c-kernel iter-4 — slot-aware prefill call mirroring
    // iter-2A/2B Generate-arm + iter-3 GenerateStream-arm shape.  Calls
    // `forward_prefill_with_soft_tokens_slot_aware` (the iter-2A landing
    // per §6.1.32 + iter-2B routing per §6.1.34) with `max_decode_tokens=0`
    // — matching `MlxModelWeights::forward_embed_last`'s call to
    // `forward_prefill(prompt_tokens, 0, gpu)` at forward_prefill.rs:2303.
    //
    // Unlike Generate / GenerateStream, the returned first-decode-token
    // is INTENTIONALLY DISCARDED: embed reads the hidden state from
    // `loaded.weights.activations.norm_out` after the prefill (the per-
    // token loop populates norm_out with the last token's RMS-normed
    // hidden state as part of its final_norm dispatch ~line 1186 in
    // forward_prefill.rs).  The first-decode-token = prefill argmax is
    // a side effect of the kernel — not part of the embed output.
    //
    // Multi-token decode-loop body wrapping (iter-B4c-kernel-iter-2-decode
    // scope on the Generate / GenerateStream surfaces) is STRUCTURALLY
    // N/A here: embed has no decode loop.
    let prefill_result = loaded.weights.forward_prefill_with_soft_tokens_slot_aware(
        prompt_tokens,
        &[], // Embed-arm: no soft-token overrides; the Embed Request
        // variant does NOT carry soft_tokens / deepstack /
        // positions_flat surfaces.
        0, // max_decode_tokens=0 — embed has no decode budget;
        // matches forward_embed_last's call shape at
        // forward_prefill.rs:2303 (the `0` triggers
        // `linear_capacity = prompt_len + 0`).
        &mut loaded.ctx,
        slot_id,
        multi_seq_kv,
        multi_seq_kv_hybrid.as_deref_mut(),
        // ADR-040 iter-2D + iter-2C (§6.1.46): dense F32 + legacy
        // 4-bit scaffold siblings.  Embed has no decode loop, so
        // these are consumed only by the prefill branches.
        multi_seq_kv_dense.as_deref_mut(),
        multi_seq_kv_mlx.as_deref_mut(),
    );

    // Read the L2-normalized hidden vector from norm_out — byte-
    // equivalent to the tail of `MlxModelWeights::forward_embed_last`
    // at forward_prefill.rs:2306-2331.  This branches on prefill_result
    // because a forward-failure leaves norm_out in an undefined state;
    // we surface the prefill error first so the operator sees the
    // load-bearing diagnostic.
    let embed_vec_result: Result<Vec<f32>> = match prefill_result {
        Ok(_first_decode_token) => {
            // Read the [hidden_size] f32 hidden state.  norm_out is
            // sized [1 row * hidden_size] — the per-token reuse of the
            // buffer means it always holds exactly one row's worth of
            // data (the last token's RMS-normed hidden state).
            let view_result = loaded.weights.activations.norm_out.as_slice().map_err(|e| {
                anyhow::anyhow!(
                    "embed_gemma4_slot_aware read norm_out: {e} \
                     (ADR-040 iter-B4c-kernel iter-4)"
                )
            });
            match view_result {
                Ok(view) => {
                    let hs = loaded.weights.hidden_size;
                    if view.len() < hs {
                        Err(anyhow::anyhow!(
                            "embed_gemma4_slot_aware: norm_out has {} f32 \
                             elements, expected at least {} (ADR-040 \
                             iter-B4c-kernel iter-4)",
                            view.len(),
                            hs
                        ))
                    } else {
                        let mut out: Vec<f32> = view[..hs].to_vec();
                        // L2 normalize so consumers can compute cosine
                        // similarity by dot product. 1e-12 floor matches
                        // the BERT-lane bert_l2_normalize_gpu epsilon
                        // (mirrors forward_embed_last:2326-2330 verbatim).
                        let norm: f32 = out.iter().map(|v| v * v).sum::<f32>().sqrt();
                        let denom = if norm < 1e-12 { 1e-12 } else { norm };
                        for v in out.iter_mut() {
                            *v /= denom;
                        }
                        Ok(out)
                    }
                }
                Err(e) => Err(e),
            }
        }
        Err(e) => Err(e.context(
            "embed_gemma4_slot_aware: forward_prefill_with_soft_tokens_slot_aware \
             (ADR-040 iter-B4c-kernel iter-4)",
        )),
    };

    // Per-slot reset at exit — leave the slot clean for the next
    // request regardless of whether the prefill succeeded or failed.
    // Belt-and-suspenders w/ the entry reset; mirrors iter-1 +
    // iter-3 exit-reset discipline.  Errors swallowed via tracing::warn
    // — entry reset on the next request will fire if this fails
    // (impossible at runtime since bounds + buffer shape are identical
    // to the entry-reset that already succeeded).
    for (layer_idx, buf) in multi_seq_kv.iter_mut().enumerate() {
        if let Err(e) = buf.reset_for_slot(slot_id) {
            tracing::warn!(
                "embed_gemma4_slot_aware: reset_for_slot at exit L{} \
                 failed (slot_id={}): {} — slot WILL be reset before next \
                 admission via the entry reset of the next call",
                layer_idx,
                slot_id.0,
                e
            );
        }
    }
    // ADR-040 iter-B4c-kernel iter-4 — exit reset on the hybrid
    // scaffold sibling.  Same swallow-on-error discipline as the HB
    // scaffold above — entry reset on the NEXT request will fire if
    // this fails (impossible at runtime since bounds + buffer shape
    // are identical to the entry-reset that already succeeded).
    if let Some(ref mut hybrid_scaffold) = multi_seq_kv_hybrid {
        for (layer_idx, buf) in hybrid_scaffold.iter_mut().enumerate() {
            if let Err(e) = buf.reset_for_slot(slot_id) {
                tracing::warn!(
                    "embed_gemma4_slot_aware: reset_for_slot (hybrid) at exit L{} \
                     failed (slot_id={}): {} — slot WILL be reset before next \
                     admission via the entry reset of the next call",
                    layer_idx,
                    slot_id.0,
                    e
                );
            }
        }
    }

    embed_vec_result
}

/// **ADR-040 iter-B4c-kernel iter-5 (2026-05-30)** — slot-aware
/// vision-aware non-streaming Gemma 4 generation against the persistent
/// multi-seq per-layer
/// [`crate::inference::models::gemma4::kv_cache::MultiSeqHbKvBuffers`]
/// scaffold (`GemmaLoadedModel.multi_seq_kv`) + the production-default
/// hybrid F16-K + TQ-HB-V sibling scaffold (`GemmaLoadedModel.
/// multi_seq_kv_hybrid`) instead of the legacy per-request inline alloc.
///
/// **Direct mirror of `generate_gemma4_once_slot_aware`** (iter-1 +
/// iter-2A + iter-2B Generate-arm lift, per §6.1.31 + §6.1.32 + §6.1.34)
/// for the [`super::engine::Request::GenerateWithSoftTokens`] worker arm
/// on the Gemma 4 architecture.  iter-1+2A+2B landed the non-streaming
/// Generate-arm lift + kernel-forward step; iter-3 landed the streaming
/// arm; iter-4 landed the Embed arm; iter-5 is the **TERMINAL Gemma 4
/// worker-arm lift** — post-iter-5 ALL FOUR Gemma 4 worker arms route
/// through the persistent multi-seq scaffolds at SlotId(N>0).
///
/// **Cross-architecture mirror of Qwen35 iter-C2d-cont-kernel iter-4
/// `engine_qwen35::generate_qwen35_once_with_soft_tokens_slot_aware`
/// per §6.1.30** — same dispatch fork shape (`slot_id != SlotId(0)`
/// predicate at the worker arm), same take-and-restore borrow pattern
/// on BOTH scaffolds, same `reset_for_slot` entry+exit discipline on
/// BOTH scaffolds, same `Result<GenerationResult>` return surface.
///
/// # Differences from iter-1+2A+2B (Generate) + iter-3 (Stream) + iter-4 (Embed)
///
/// | Dimension | iter-1+2A+2B (Generate) | iter-3 (Stream) | iter-4 (Embed) | iter-5 (SoftTokens) |
/// |---|---|---|---|---|
/// | Result surface | `Result<GenerationResult>` (synchronous) | SSE event channel | `Result<Vec<f32>>` | `Result<GenerationResult>` (synchronous) |
/// | Soft tokens | `&[]` (empty) | `&[]` (vision deferred to iter-5) | `&[]` (no soft tokens on Embed) | **`soft_tokens` carried** through to the slot-aware prefill kernel |
/// | Deepstack / positions_flat | N/A | N/A | N/A | **N/A** — Gemma 4 does not consume deepstack / 3D positions (those are Qwen3-VL specific; non-slot-aware sibling at engine.rs:6144-6155 falls back to soft-token-only entry) |
/// | iter-2-decode sub-deferral | APPLIES — typed `CapabilityUnsupported` after prefill Ok branch | APPLIES — typed SSE Error event after prefill Ok branch | **N/A** — embed has no decode loop | APPLIES — typed `CapabilityUnsupported` after prefill Ok branch (same shape as Generate-arm) |
///
/// # Structural parallels with iter-1+2A+2B (Generate) + iter-4 (Embed)
///
/// 1. Bounds-checks `slot_id` against `multi_seq_kv[0].n_seqs` (bounds-
///    first per A2b §6.1.23 iter-1.5 cfa-finding-F5 ordering); surfaces
///    typed `anyhow::Error` with `capability_unsupported:` prefix + cite.
/// 2. Calls `MultiSeqHbKvBuffers::reset_for_slot(slot_id)` at entry on
///    every per-layer buffer — zeros the per-seq cursor for `slot_id`
///    only (other slots untouched).
/// 3. Mirrors entry-reset on the production-default hybrid scaffold
///    sibling (`multi_seq_kv_hybrid`) when `Some(_)` (HF2Q_HYBRID_KV=1
///    per H10 §6.1.11 default).
/// 4. Calls `loaded.weights.forward_prefill_with_soft_tokens_slot_aware(..)`
///    (the iter-2A landing per §6.1.32 + iter-2B routing per §6.1.34)
///    with the caller-supplied `soft_tokens` + `max_decode_tokens =
///    params.max_tokens.max(1)`, threading `slot_id` + both scaffolds.
/// 5. Surfaces a typed `iter-B4c-kernel-iter-2-decode` sub-deferral on
///    the prefill Ok branch — mirrors the Generate-arm IIFE discipline
///    at §6.1.32 (the multi-token decode-loop body wrapping requires
///    `forward_decode` to thread `slot_id`, which is iter-2-decode
///    scope).
/// 6. Calls `reset_for_slot(slot_id)` at exit on BOTH scaffolds — belt-
///    and-suspenders with the entry reset (mirror of iter-1's exit
///    discipline).
///
/// **Per-slot byte-equivalence at SlotId(0)** (H116 pin): the worker
/// arm's `handle.slot_id != SlotId(0)` predicate short-circuits AT the
/// worker arm — `generate_gemma4_once_with_soft_tokens_slot_aware` is
/// NEVER called for SlotId(0).  Both SerialFifo (always SlotId(0)) and
/// SlotAware + SlotId(0) route through the existing `generate_once_with_soft_tokens`
/// dispatch verbatim, preserving the H1/H2/H23/H41/H44/H77/H104/H110
/// byte-equivalence chain.
///
/// # Empty soft-token fall-through
///
/// When `soft_tokens.is_empty()`, the body is identity over
/// `generate_gemma4_once_slot_aware` (the text-only Generate-arm
/// slot-aware fn).  Mirrors the non-slot-aware sibling
/// `generate_once_with_soft_tokens` behavior — when no soft-token
/// overrides are present, the function reduces to the text-only path.
/// This discipline matches Qwen35 iter-4's empty-soft early-return at
/// engine_qwen35.rs:5041-5050.
///
/// # Errors
/// - `prompt_tokens.is_empty()` (matches `forward_prefill_with_soft_tokens_slot_aware`).
/// - `slot_id.0 >= multi_seq_kv[0].n_seqs` (bounds-first).
/// - `reset_for_slot` failure propagates with `iter-B4c-kernel iter-5`
///   context.
/// - `forward_prefill_with_soft_tokens_slot_aware` failure propagates.
/// - `iter-B4c-kernel-iter-2-decode` typed `CapabilityUnsupported` on
///   the prefill Ok branch (same shape as the Generate-arm).
#[allow(clippy::too_many_arguments)]
fn generate_gemma4_once_with_soft_tokens_slot_aware(
    loaded: &mut GemmaLoadedModel,
    prompt_tokens: &[u32],
    soft_tokens: &[SoftTokenInjection<'_>],
    params: &SamplingParams,
    registration: Option<&super::registry::ModelRegistration>,
    multi_seq_kv: &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqHbKvBuffers>,
    // ADR-040 iter-B4c-kernel iter-5 (2026-05-30) — production-default
    // hybrid F16-K + TQ-HB-V scaffold sibling param.  `Option<>` because
    // iter-C2c-cont provisions this field IFF the hybrid env-gate is
    // ON (DEFAULT per H10 §6.1.11).  Threaded verbatim through to the
    // model fn — same shape as the Generate-arm sibling per iter-2B,
    // the GenerateStream-arm sibling per iter-3, and the Embed-arm
    // sibling per iter-4.
    //
    // `mut` binding so the orchestrator body can do entry+exit
    // `reset_for_slot` via `if let Some(ref mut _) = multi_seq_kv_hybrid`
    // AND pass an `as_deref_mut()` reborrow to the model fn call below.
    mut multi_seq_kv_hybrid: Option<
        &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqHybridKvBuffers>,
    >,
    // ADR-040 iter-B4c-kernel iter-2D + iter-2C (§6.1.46) — dense F32 +
    // legacy 4-bit scaffold siblings.  Threaded as Option<&mut>
    // identically to iter-3/4 worker arms.
    mut multi_seq_kv_dense: Option<
        &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqDenseKvBuffers>,
    >,
    mut multi_seq_kv_mlx: Option<
        &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqMlxKvCache>,
    >,
    slot_id: SlotId,
) -> Result<GenerationResult> {
    // Empty soft-token slice → identity over the text-only slot-aware
    // path.  Mirrors the non-slot-aware sibling `generate_once_with_soft_tokens`
    // shape (which itself reduces to `generate_once` when soft-tokens
    // are absent) + Qwen35 iter-4's empty-soft early-return at
    // engine_qwen35.rs:5041-5050.
    //
    // This discipline is structurally necessary: the SoftTokens-arm
    // surface is the ONLY worker arm where the request channel carries
    // a soft-token vec, but a text-only request that happens to route
    // through this arm (e.g. chat handler routing edge cases) MUST
    // produce byte-identical output to the Generate-arm — calling the
    // Generate-arm slot-aware fn directly preserves that invariant.
    if soft_tokens.is_empty() {
        return generate_gemma4_once_slot_aware(
            loaded,
            prompt_tokens,
            params,
            registration,
            multi_seq_kv,
            multi_seq_kv_hybrid,
            // ADR-040 iter-2D + iter-2C (§6.1.46) — early-return forwards
            // the dense F32 + legacy 4-bit scaffold siblings verbatim.
            multi_seq_kv_dense,
            multi_seq_kv_mlx,
            slot_id,
        );
    }

    anyhow::ensure!(
        !prompt_tokens.is_empty(),
        "generate_gemma4_once_with_soft_tokens_slot_aware: empty prompt_tokens"
    );
    anyhow::ensure!(
        !multi_seq_kv.is_empty(),
        "generate_gemma4_once_with_soft_tokens_slot_aware: multi_seq_kv is empty \
         (C2c spawn-arm invariant: provision_multi_seq_kv_for_slot_aware \
          must produce one entry per layer; ADR-040 §6.1.21)"
    );
    // Bounds-first per A2b §6.1.23 iter-1.5 cfa-finding-F5 ordering.
    // Use the first layer's n_seqs as the canonical bound — A3a
    // construction guarantees every per-layer entry has the same
    // n_seqs (provisioned with max_slots).
    let n_seqs = multi_seq_kv[0].n_seqs;
    anyhow::ensure!(
        slot_id.0 < n_seqs,
        "generate_gemma4_once_with_soft_tokens_slot_aware: SlotOutOfRange slot={} \
         max_slots={} (ADR-040 iter-B4c-kernel iter-5)",
        slot_id.0,
        n_seqs,
    );

    // Per-slot reset at entry — the persistent cache may carry stale
    // bytes from a prior request on this slot.  `reset_for_slot` zeros
    // the per-seq cursor for `slot_id` on EVERY per-layer entry; K/V
    // packed + norms bytes are cursor-masked.
    //
    // Mirror of iter-1 Generate-arm + iter-3 GenerateStream-arm +
    // iter-4 Embed-arm entry-reset discipline.
    for (layer_idx, buf) in multi_seq_kv.iter_mut().enumerate() {
        buf.reset_for_slot(slot_id).map_err(|e| {
            anyhow::anyhow!(
                "generate_gemma4_once_with_soft_tokens_slot_aware: reset_for_slot at \
             entry L{layer_idx}: {e} (ADR-040 iter-B4c-kernel iter-5)"
            )
        })?;
    }
    // ADR-040 iter-B4c-kernel iter-5 — entry reset on the hybrid
    // scaffold sibling.  Mirrors the HB scaffold entry-reset discipline
    // above for the production-default regime.
    if let Some(ref mut hybrid_scaffold) = multi_seq_kv_hybrid {
        for (layer_idx, buf) in hybrid_scaffold.iter_mut().enumerate() {
            buf.reset_for_slot(slot_id).map_err(|e| {
                anyhow::anyhow!(
                    "generate_gemma4_once_with_soft_tokens_slot_aware: reset_for_slot \
                 at entry (hybrid) L{layer_idx}: {e} (ADR-040 iter-B4c-kernel iter-5)"
                )
            })?;
        }
    }

    // ADR-040 iter-B4c-kernel iter-5 — slot-aware vision-aware prefill
    // call.  Threads the caller-supplied `soft_tokens` through to
    // `forward_prefill_with_soft_tokens_slot_aware` (the iter-2A
    // landing per §6.1.32 + iter-2B routing per §6.1.34); this is the
    // load-bearing difference from the Generate-arm iter-1+2A+2B body
    // which always passes `&[]`.
    //
    // Multi-token decode-loop body wrapping is iter-B4c-kernel-iter-2-decode
    // scope (same sub-deferral the Generate / GenerateStream arms
    // surface).  Mirror of the Generate-arm IIFE discipline at
    // §6.1.32 — until iter-2-decode lands, we surface typed
    // `CapabilityUnsupported` on the prefill Ok branch.
    let max_decode_tokens = params.max_tokens.max(1);
    // ADR-040 iter-B4c-kernel iter-2-decode-A (2026-05-30) — SoftTokens-arm
    // (vision-aware) decode-loop body landed (greedy fast-path).
    //
    // ADR-040 iter-B4c-kernel iter-2-decode-C (2026-05-30) — FULL sampler/
    // grammar/stop-strings/logprobs/reasoning-text surface lands.
    // Identical to Generate-arm full surface — the SoftTokens-vs-Generate
    // difference is fully consumed by the prefill call's `soft_tokens`
    // parameter; the post-prefill decode body's sampler / grammar /
    // logprobs / stop-strings / reasoning shape is identical.
    let kernel_forward_result: Result<GenerationResult> = (|| -> Result<GenerationResult> {
        let prefill_started = Instant::now();
        let first_decode_token = loaded.weights.forward_prefill_with_soft_tokens_slot_aware(
            prompt_tokens,
            soft_tokens, // SoftTokens-arm: vision-aware soft-token
            // overrides threaded through — this is the
            // load-bearing difference from iter-1+2A+2B
            // (Generate-arm passes `&[]`).
            max_decode_tokens,
            &mut loaded.ctx,
            slot_id,
            multi_seq_kv,
            multi_seq_kv_hybrid.as_deref_mut(),
            // ADR-040 iter-2D + iter-2C (§6.1.46).
            multi_seq_kv_dense.as_deref_mut(),
            multi_seq_kv_mlx.as_deref_mut(),
        )?;
        let prefill_duration = prefill_started.elapsed();

        // ── Sampler / grammar / logprobs config (mirror of Generate-arm).
        let sample_logits = params.temperature > 0.0
            || params.top_k > 0
            || params.top_p < 1.0
            || params.repetition_penalty != 1.0
            || !params.logit_bias.is_empty()
            || params.grammar.is_some()
            || params.logprobs;
        let sampler_params = if sample_logits {
            Some(SamplerParams {
                temperature: params.temperature as f64,
                top_p: params.top_p as f64,
                top_k: params.top_k,
                min_p: 0.0,
                repetition_penalty: effective_repetition_penalty(params),
                max_tokens: params.max_tokens,
            })
        } else {
            None
        };

        let mut grammar_runtime: Option<super::grammar::GrammarRuntime> =
            match params.grammar.as_ref() {
                Some(g) => {
                    let start_rule_id = g
                        .rule_id("root")
                        .ok_or_else(|| anyhow::anyhow!("grammar has no root rule"))?;
                    let mut rt = super::grammar::GrammarRuntime::new(g.clone(), start_rule_id)
                        .ok_or_else(|| anyhow::anyhow!("grammar runtime init failed"))?;
                    if matches!(params.grammar_kind, GrammarKind::ToolCallBodyAuto) {
                        rt.set_awaiting_trigger(true);
                    }
                    Some(rt)
                }
                None => None,
            };
        let token_bytes_ref: Option<&[Vec<u8>]> = params.token_bytes.as_deref().map(|v| &v[..]);
        let mut tc_splitter_ns: Option<super::registry::ToolCallSplitter> =
            registration.and_then(super::registry::ToolCallSplitter::from_registration);

        let want_logprobs = params.logprobs;
        let mut logprobs_acc: Option<Vec<f32>> = if want_logprobs {
            Some(Vec::with_capacity(params.max_tokens))
        } else {
            None
        };

        let mut next_token = if sample_logits {
            let sp = sampler_params.as_ref().expect("sample_logits gate");
            let mut logits: Vec<f32> = loaded.weights.logits_view()?.to_vec();
            if !params.logit_bias.is_empty() {
                let v = logits.len();
                for (&id, &bias) in &params.logit_bias {
                    let idx = id as usize;
                    if idx < v {
                        logits[idx] += bias;
                    }
                }
            }
            if let (Some(rt), Some(tb)) = (grammar_runtime.as_ref(), token_bytes_ref) {
                super::grammar::mask::mask_invalid_tokens(rt, tb, &mut logits);
            }
            let (tok, lp_opt) = if want_logprobs {
                let (t, lp) = sampler_pure::sample_token_with_logprob(&mut logits, sp, &[]);
                (t, Some(lp))
            } else {
                (sampler_pure::sample_token(&mut logits, sp, &[]), None)
            };
            if let (Some(acc), Some(lp_val)) = (logprobs_acc.as_mut(), lp_opt) {
                acc.push(lp_val);
            }
            if let (Some(rt), Some(tb)) = (grammar_runtime.as_mut(), token_bytes_ref) {
                let bytes = tb.get(tok as usize).map(|v| v.as_slice()).unwrap_or(&[]);
                if !bytes.is_empty() {
                    rt.accept_bytes(bytes);
                }
            }
            tok
        } else {
            first_decode_token
        };

        let mut splitter = registration.filter(|r| r.has_reasoning()).and_then(|r| {
            super::registry::make_reasoning_splitter(r, params.reasoning_forced_open)
        });
        let reasoning_enabled = splitter.is_some();
        let mut reasoning_token_count: usize = 0;

        let decode_started = Instant::now();
        let mut generated_tokens: Vec<u32> = Vec::with_capacity(max_decode_tokens);
        let mut decoded_text = String::new();

        let first_fragment = loaded
            .tokenizer
            .decode(&[next_token], false)
            .unwrap_or_default();
        decoded_text.push_str(&first_fragment);
        if let Some(sp) = splitter.as_mut() {
            let _ = sp.feed(&first_fragment);
            if sp.in_reasoning() {
                reasoning_token_count += 1;
            }
        }
        if let Some(tcs) = tc_splitter_ns.as_mut() {
            let events = tcs.feed(&first_fragment);
            if let Some(rt) = grammar_runtime.as_mut() {
                if events
                    .iter()
                    .any(|e| matches!(e, super::registry::ToolCallEvent::ToolCallOpen))
                {
                    rt.trigger();
                }
            }
        }

        let mut finish_reason: &'static str = "length";

        if loaded.eos_token_ids.contains(&next_token) {
            finish_reason = "stop";
        } else if hit_stop_string(&decoded_text, &params.stop_strings) {
            finish_reason = "stop";
            strip_trailing_stop(&mut decoded_text, &params.stop_strings);
        } else {
            generated_tokens.push(next_token);
            for _ in 1..max_decode_tokens {
                let pos = prompt_tokens.len() + generated_tokens.len() - 1;
                let mut p: Option<crate::inference::models::gemma4::profile::TokenProfile> = None;
                let greedy_token = loaded.weights.forward_decode_slot_aware(
                    next_token,
                    pos,
                    &mut loaded.ctx,
                    &mut p,
                    slot_id,
                    multi_seq_kv,
                    multi_seq_kv_hybrid.as_deref_mut(),
                    // ADR-040 iter-2-decode-D (§6.1.46) — dense F32 +
                    // legacy 4-bit decode-side scaffold siblings.
                    multi_seq_kv_dense.as_deref_mut(),
                    multi_seq_kv_mlx.as_deref_mut(),
                )?;

                next_token = if sample_logits {
                    let sp = sampler_params.as_ref().expect("sample_logits gate");
                    let mut logits: Vec<f32> = loaded.weights.logits_view()?.to_vec();
                    if !params.logit_bias.is_empty() {
                        let v = logits.len();
                        for (&id, &bias) in &params.logit_bias {
                            let idx = id as usize;
                            if idx < v {
                                logits[idx] += bias;
                            }
                        }
                    }
                    if let (Some(rt), Some(tb)) = (grammar_runtime.as_ref(), token_bytes_ref) {
                        super::grammar::mask::mask_invalid_tokens(rt, tb, &mut logits);
                    }
                    let (tok, lp_opt) = if want_logprobs {
                        let (t, lp) = sampler_pure::sample_token_with_logprob(
                            &mut logits,
                            sp,
                            &generated_tokens,
                        );
                        (t, Some(lp))
                    } else {
                        (
                            sampler_pure::sample_token(&mut logits, sp, &generated_tokens),
                            None,
                        )
                    };
                    if let (Some(acc), Some(lp_val)) = (logprobs_acc.as_mut(), lp_opt) {
                        acc.push(lp_val);
                    }
                    if let (Some(rt), Some(tb)) = (grammar_runtime.as_mut(), token_bytes_ref) {
                        let bytes = tb.get(tok as usize).map(|v| v.as_slice()).unwrap_or(&[]);
                        if !bytes.is_empty() {
                            rt.accept_bytes(bytes);
                        }
                    }
                    tok
                } else {
                    greedy_token
                };

                if loaded.eos_token_ids.contains(&next_token) {
                    finish_reason = "stop";
                    break;
                }
                generated_tokens.push(next_token);
                let fragment = loaded
                    .tokenizer
                    .decode(&[next_token], false)
                    .unwrap_or_default();
                decoded_text.push_str(&fragment);
                if let Some(sp) = splitter.as_mut() {
                    let _ = sp.feed(&fragment);
                    if sp.in_reasoning() {
                        reasoning_token_count += 1;
                    }
                }
                if let Some(tcs) = tc_splitter_ns.as_mut() {
                    let events = tcs.feed(&fragment);
                    if let Some(rt) = grammar_runtime.as_mut() {
                        if events
                            .iter()
                            .any(|e| matches!(e, super::registry::ToolCallEvent::ToolCallOpen))
                        {
                            rt.trigger();
                        }
                    }
                }
                if hit_stop_string(&decoded_text, &params.stop_strings) {
                    finish_reason = "stop";
                    strip_trailing_stop(&mut decoded_text, &params.stop_strings);
                    break;
                }
                if grammar_runtime.as_ref().is_some_and(|rt| rt.is_dead()) {
                    finish_reason = "stop";
                    generated_tokens.pop();
                    decoded_text = loaded
                        .tokenizer
                        .decode(&generated_tokens, false)
                        .unwrap_or_default();
                    break;
                }
            }
        }
        let decode_duration = decode_started.elapsed();

        let (content, reasoning_text) = match registration {
            Some(reg) if reg.has_reasoning() => super::registry::split_full_output_forced(
                reg,
                &decoded_text,
                params.reasoning_forced_open,
            ),
            _ => (decoded_text, None),
        };

        Ok(GenerationResult {
            text: content,
            reasoning_text,
            prompt_tokens: prompt_tokens.len(),
            completion_tokens: generated_tokens.len(),
            reasoning_tokens: if reasoning_enabled && reasoning_token_count > 0 {
                Some(reasoning_token_count)
            } else {
                None
            },
            finish_reason,
            prefill_duration,
            decode_duration,
            cached_tokens: 0,
            logprobs: logprobs_acc,
        })
    })();

    // Per-slot reset at exit — leave the slot clean for the next
    // request regardless of whether the kernel-forward sub-deferral
    // returned Ok or Err.  Belt-and-suspenders w/ the entry reset;
    // mirrors iter-1 / iter-3 / iter-4 exit-reset discipline.  Errors
    // swallowed via tracing::warn — entry reset on the next request
    // will fire if this fails.
    for (layer_idx, buf) in multi_seq_kv.iter_mut().enumerate() {
        if let Err(e) = buf.reset_for_slot(slot_id) {
            tracing::warn!(
                "generate_gemma4_once_with_soft_tokens_slot_aware: reset_for_slot \
                 at exit L{} failed (slot_id={}): {} — slot WILL be reset before \
                 next admission via the entry reset of the next call",
                layer_idx,
                slot_id.0,
                e
            );
        }
    }
    // ADR-040 iter-B4c-kernel iter-5 — exit reset on the hybrid
    // scaffold sibling.  Same swallow-on-error discipline as the HB
    // scaffold above.
    if let Some(ref mut hybrid_scaffold) = multi_seq_kv_hybrid {
        for (layer_idx, buf) in hybrid_scaffold.iter_mut().enumerate() {
            if let Err(e) = buf.reset_for_slot(slot_id) {
                tracing::warn!(
                    "generate_gemma4_once_with_soft_tokens_slot_aware: reset_for_slot \
                     (hybrid) at exit L{} failed (slot_id={}): {} — slot WILL be \
                     reset before next admission via the entry reset of the next call",
                    layer_idx,
                    slot_id.0,
                    e
                );
            }
        }
    }
    kernel_forward_result
}

// `infer_quant_type_from_gguf` was relocated to
// `crate::serve::load_info::infer_quant_label` per ADR-018 C1.  The
// previous 27-LOC body was byte-identical to (and the call site here
// shared an algorithm with) the equivalent body in
// `engine_qwen35.rs:246-272`; both call sites now route through the
// promoted helper.

/// Outcome of `finalize_streaming_tool_state` — tells the streaming
/// driver whether to proceed to `Done`, abort silently (client gone),
/// or skip `Done` because a structured `Error` event has already been
/// emitted by the helper. Wave 2.8 W-θ HIGH-1.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum FinalizeStreamingAction {
    /// Drain succeeded (or was a no-op). Caller proceeds to emit Done.
    Continue,
    /// `events.blocking_send` returned Err while emitting a tail Content
    /// delta. SSE receiver is gone; caller should abort silently (no Done).
    ClientDropped,
    /// Helper emitted a `GenerationEvent::Error` (Constrained mid-call
    /// truncation or no-call). Caller MUST skip Done — the SSE encoder
    /// produces the final error chunk on receipt of Error.
    ErrorEmitted,
}

/// Drain the tool-call splitter tail at end-of-stream and then enforce the
/// grammar-active-policy safety nets before the streaming `Done` event.
///
/// Wave 2.8 W-θ HIGH-1 — streaming companion to the non-streaming
/// defensive 500 in `handlers.rs:410-444` (commit da545d5).
/// Wave 3 W-B2 — `AutoLazyGrammar` joins `Constrained` in the loud-error
/// branch (mid-call truncation only; no-call check stays Constrained-only
/// because Auto-lazy explicitly allows the model to emit zero calls).
///
/// Behaviour matrix (`policy` = `tool_call_policy`):
/// ```text
///                       │ Auto (no grammar)              │ AutoLazyGrammar / Constrained
/// ─────────────────────┼─────────────────────────────────┼────────────────────────────────
/// finish() = Content   │ emit Content                    │ emit Content
/// finish() = TC-text   │ emit Content (open-marker re-   │ emit GenerationEvent::Error
///                       │   prepended for clarity)        │   "tool_call_truncated_under_constrained"
///                       │                                 │   → ErrorEmitted
/// post-drain no call   │ no action                       │ Constrained ONLY: emit Error
///                       │                                 │   "tool_call_no_call_under_constrained"
///                       │                                 │ AutoLazyGrammar: no action
///                       │                                 │   (Auto explicitly allows no-call)
/// ```
///
/// **Why mid-call truncation errors under both Constrained AND
/// AutoLazyGrammar**: in either case the grammar is active inside the
/// tool-call body. A truncation past the open marker but before the
/// close marker means decoding stopped mid-grammar — the runtime is
/// neither `is_accepted()` nor `is_dead()` and the body bytes captured
/// so far cannot be parsed into a tool call. Same regression signature
/// as Constrained truncation; same loud-error promotion.
///
/// **Why no-call check stays Constrained-only**: Auto explicitly permits
/// the model to emit zero tool calls (preamble freedom — the whole
/// point of lazy grammar). A streaming run that ended without ever
/// firing `ToolCallOpen` is the legitimate Auto-no-call path, NOT a
/// regression. Required/Function on the other hand mandate at least
/// one call (the eager grammar root accepts only `OneOrMoreCalls`),
/// so a no-call run there means max_tokens cut the call mid-emission
/// or the grammar emitter has a bug.
///
/// Extracted from the inline finalize block so the audit-driver test for
/// HIGH-1 can exercise this exact code path. See
/// `finalize_streaming_tool_state_tests` below.
fn finalize_streaming_tool_state(
    tool_splitter: Option<&mut super::registry::ToolCallSplitter>,
    policy: ToolCallPolicy,
    saw_tool_call: bool,
    registration: Option<&super::registry::ModelRegistration>,
    completion_tokens: usize,
    accumulated_text_len: usize,
    events: &EventSink<'_>,
) -> FinalizeStreamingAction {
    use super::sse::{DeltaKind, GenerationEvent};

    // Wave 3 W-B2: mid-call truncation fires for any policy carrying an
    // active body grammar (Constrained from byte 0, AutoLazyGrammar from
    // ToolCallOpen onwards). The single source of truth lives in
    // `ToolCallPolicy::enforces_body_grammar`.
    let body_grammar_active = policy.enforces_body_grammar();
    // Wave 3 W-B2: no-call check stays Constrained-only; AutoLazyGrammar
    // explicitly allows the model to emit zero calls (preamble freedom).
    let policy_constrained = matches!(policy, ToolCallPolicy::Constrained);

    if let Some(tcs) = tool_splitter {
        if let Some(ev) = tcs.finish() {
            match ev {
                super::registry::ToolCallEvent::Content(t) => {
                    if !t.is_empty()
                        && events
                            .blocking_send(GenerationEvent::Delta {
                                kind: DeltaKind::Content,
                                text: t,
                            })
                            .is_err()
                    {
                        return FinalizeStreamingAction::ClientDropped;
                    }
                }
                super::registry::ToolCallEvent::ToolCallText(t) => {
                    if body_grammar_active {
                        // HIGH-1 streaming companion to da545d5 + Wave 3
                        // W-B2: emit a structured error event INSTEAD of
                        // silently re-emitting the residual as Content.
                        // Fires for Constrained AND AutoLazyGrammar.
                        let policy_label = match policy {
                            ToolCallPolicy::Constrained => "required/function",
                            ToolCallPolicy::AutoLazyGrammar => "auto (lazy grammar active)",
                            ToolCallPolicy::Auto => {
                                unreachable!("body_grammar_active gate")
                            }
                        };
                        tracing::error!(
                            residual_len = t.len(),
                            policy = policy_label,
                            "tool_call_truncated_under_constrained: streaming \
                             ended mid-tool-call (open marker observed, no close \
                             marker) under tool_choice={}; per-model body \
                             grammar should have prevented this",
                            policy_label
                        );
                        let _ = events.blocking_send(GenerationEvent::Error(
                            "tool_call_truncated_under_constrained".into(),
                        ));
                        return FinalizeStreamingAction::ErrorEmitted;
                    }
                    // Auto (no grammar): legacy behaviour — emit residual
                    // body as Content with the literal open marker
                    // re-prepended for diagnostic clarity (the splitter
                    // swallowed the open marker when it flipped state).
                    let prefix = registration.and_then(|r| r.tool_open).unwrap_or("");
                    let fallback = format!("{prefix}{t}");
                    if !fallback.is_empty()
                        && events
                            .blocking_send(GenerationEvent::Delta {
                                kind: DeltaKind::Content,
                                text: fallback,
                            })
                            .is_err()
                    {
                        return FinalizeStreamingAction::ClientDropped;
                    }
                }
                super::registry::ToolCallEvent::ToolCallOpen
                | super::registry::ToolCallEvent::ToolCallClose => {
                    // unreachable: finish() never emits Open/Close.
                }
            }
        }
    }

    // Post-drain no-call check — streaming companion to handlers.rs:410-444.
    // Constrained ONLY (Required/Function): grammar root mandates
    // OneOrMoreCalls; a no-call run is a regression. AutoLazyGrammar
    // explicitly permits no-call (the whole point of lazy grammar is
    // preamble freedom + optional emission).
    if policy_constrained && !saw_tool_call {
        tracing::error!(
            completion_tokens = completion_tokens,
            text_len = accumulated_text_len,
            "tool_call_no_call_under_constrained: streaming ended with zero \
             tool calls under tool_choice=required/function; eager grammar \
             should have prevented this — either max_tokens cut a call \
             mid-emission or the grammar emitter has a bug"
        );
        let _ = events.blocking_send(GenerationEvent::Error(
            "tool_call_no_call_under_constrained".into(),
        ));
        return FinalizeStreamingAction::ErrorEmitted;
    }

    FinalizeStreamingAction::Continue
}

/// Wave 3 W-B3 — T2.3 incremental tool-call argument streaming.
///
/// Per OpenAI Chat Completions streaming spec, `delta.tool_calls[N].function.arguments`
/// is a *string accumulator* on the client side — clients append each arg-delta to
/// the previous, then `JSON.parse(accumulated)` once the chunk with `finish_reason ==
/// "tool_calls"` arrives. Pre-W-B3, the streaming engine accumulated the entire
/// per-family body (Gemma `<|tool_call>...<tool_call|>`, Qwen `<tool_call>...</tool_call>`)
/// into `tool_call_body`, then on `ToolCallClose` parsed it and emitted a SINGLE
/// arguments delta carrying the full JSON. Spec-valid, but a UI cannot show
/// progressive tool-call args while the model is still emitting them.
///
/// The emitter wires in three places inside the `route_content` closure:
///
///   - **`ToolCallOpen`**: construct a fresh `ToolCallStreamEmitter` for this call.
///   - **`ToolCallText(t)`**: after appending `t` to `tool_call_body`, call
///     `emitter.advance(body, events)` which:
///       1. Emits the **first chunk** as soon as the function name is parseable
///          from the body prefix (`{index, id, type:"function", function:{name}}`).
///       2. After the first chunk fires, emits the JSON args opening `{` as the
///          first `arguments` delta — clients begin accumulating the JSON string.
///       3. For each newly-closed kv pair (Gemma: top-level `,` or `}`; Qwen:
///          `</parameter>` block boundary), emits `,"key":<jsonval>` as a fresh
///          `arguments` delta (no leading `,` for the first kv).
///   - **`ToolCallClose`**: call `emitter.finalize(body, events, ...)` which:
///       - On the happy path (incremental emission started + body re-parses),
///         emits any tail kvs that the streaming scanner missed (last one before
///         the closer) and the closing `}`. `tc_index` increments here.
///       - On the fallback path (incremental emission never started — body parsed
///         OK but arrived in one fragment short of name extraction; OR the family
///         has no streaming converter; OR partial extraction failed mid-stream),
///         delegates to `emit_streaming_tool_call_close` which preserves the
///         pre-W-B3 close-buffered shape AND the policy-enforced loud-error
///         branches.
///
/// # Tail-parser design
///
/// The streaming scanner walks `body[scan_cursor..]` and extracts kv pairs at
/// JSON-syntactically-meaningful boundaries — closed string values, terminated
/// bare numerics, closed `<parameter>` blocks. It NEVER emits mid-string or
/// mid-key. The grammar runtime (eager from W-η for Constrained / lazy from
/// W-B2 for AutoLazyGrammar) physically guarantees the body bytes are
/// well-formed at every prefix the scanner inspects, so partial-prefix parse
/// failures inside the scanner are a grammar-engine bug — surfaced by leaving
/// `kvs_emitted` short, which forces the close-time `finalize` to fall through
/// to the close-buffered path and trigger the existing loud-error branch.
///
/// # Why NOT add a "speculative close + diff" approach
///
/// Considered: append the family's expected closer to `body`, re-parse with
/// `parse_tool_call_body`, diff against the last successful args-JSON, emit
/// the new tail. Rejected because:
///   - String values would emit STALE partials. `body = "call:f{loc:<|\"|>San Fra"`
///     speculatively closes to `{"loc":"San Fra"}`; clients would see `"San Fra"`
///     which then disagrees with the final `"San Francisco"`. OpenAI clients
///     concatenate without dedup — they would receive `"San FraSan Francisco"`.
///   - The closed-kv scanner is structurally simpler AND emits only at boundaries
///     that JSON treats as values committed (the previous kv + comma terminator).
///
/// # Backward compatibility
///
/// Single-chunk emission still works: clients that don't care about progressive
/// UI updates simply concatenate any number of `arguments` deltas and JSON-parse
/// the result. The spec does not bound how many deltas a server emits per call.
/// The first-chunk-has-name + finish_reason="tool_calls"-on-terminal contract
/// is preserved on both shapes.
struct ToolCallStreamEmitter {
    /// `gemma4` / `qwen35` / unknown. Unknown families skip the streaming path
    /// (the close-time fallback handles them).
    family: Option<&'static str>,
    /// `delta.tool_calls[N].index` — pre-incremented from the per-stream counter
    /// at construction. Stable across all chunks for THIS call.
    index: usize,
    /// Synthesized opaque identifier emitted in the first chunk. Cached so
    /// `finalize` can reuse it on the close-buffered fallback if the streaming
    /// path never fired (we never emitted the first chunk under that branch
    /// either, so the cached id stays unused — kept for symmetry).
    id: String,
    /// Whether the first chunk (id+type+name) has been emitted. Latched true.
    name_emitted: bool,
    /// Whether the args opening `{` has been emitted as the first arguments
    /// delta. Latched true after `name_emitted` flips and the first kv-emit
    /// or `finalize` runs (clients need the `{` before any kv content).
    args_open_emitted: bool,
    /// Number of top-level kv pairs already emitted to the client. Drives the
    /// leading-comma decision (no comma for the first kv).
    kvs_emitted: usize,
    /// Byte cursor into `tool_call_body` — bytes < cursor have been scanned
    /// for kv-emission. Bytes >= cursor are unscanned (may contain a
    /// completed-but-not-yet-emitted kv OR a partial kv in progress).
    scan_cursor: usize,
}

impl ToolCallStreamEmitter {
    /// Construct an emitter for a tool-call span starting at `tc_index`. The
    /// caller is responsible for passing the SAME `tc_index` to `finalize`'s
    /// fallback path so the close-buffered shape stays aligned when the
    /// streaming converter declines.
    fn new(family: Option<&'static str>, tc_index: usize) -> Self {
        let id = format!(
            "call_hf2q_{:016x}",
            std::time::SystemTime::now()
                .duration_since(std::time::UNIX_EPOCH)
                .map(|d| d.as_nanos() as u64)
                .unwrap_or(0)
                ^ (tc_index as u64).wrapping_mul(0x9e3779b97f4a7c15)
        );
        Self {
            family,
            index: tc_index,
            id,
            name_emitted: false,
            args_open_emitted: false,
            kvs_emitted: 0,
            scan_cursor: 0,
        }
    }

    /// Advance the emitter against the current `body` after a `ToolCallText`
    /// fragment has been appended. Emits any newly-extractable name / kv
    /// fragments. Idempotent on repeated calls with the same body — safe to
    /// invoke even when no new bytes arrived.
    fn advance(&mut self, body: &str, events: &EventSink<'_>) -> Result<(), ()> {
        use super::sse::GenerationEvent;

        // Step 1: emit the name + opening chunk if not yet done.
        if !self.name_emitted {
            let name = match self.family {
                Some("gemma4") => extract_gemma4_name_prefix(body),
                Some("qwen35") => extract_qwen35_name_prefix(body),
                _ => None,
            };
            let Some((name, header_end)) = name else {
                return Ok(());
            };
            // First chunk: index + id + type + name. arguments omitted —
            // clients see `function.name` complete on chunk 1 per spec.
            if events
                .blocking_send(GenerationEvent::ToolCallDelta {
                    index: self.index,
                    id: Some(self.id.clone()),
                    call_type: Some("function".into()),
                    name: Some(name),
                    arguments: None,
                })
                .is_err()
            {
                return Err(());
            }
            self.name_emitted = true;
            self.scan_cursor = header_end;
        }

        // Step 2: emit the opening `{` of the args object (once).
        if !self.args_open_emitted {
            if events
                .blocking_send(GenerationEvent::ToolCallDelta {
                    index: self.index,
                    id: None,
                    call_type: None,
                    name: None,
                    arguments: Some("{".into()),
                })
                .is_err()
            {
                return Err(());
            }
            self.args_open_emitted = true;
        }

        // Step 3: scan body[scan_cursor..] for newly-closed kv pairs.
        match self.family {
            Some("gemma4") => self.scan_emit_gemma4_kvs(body, events)?,
            Some("qwen35") => self.scan_emit_qwen35_kvs(body, events)?,
            _ => {}
        }
        Ok(())
    }

    /// Scan-and-emit closed Gemma 4 kvs from `body[scan_cursor..]`. A kv is
    /// "closed" when we observe its terminator at top level — `,` for
    /// non-final kvs, `}` for the final one. We emit only on `,` boundaries
    /// during streaming; the trailing `}` is finalize's job (the last kv
    /// before `}` may not yet be in the body, so we can't speculate).
    fn scan_emit_gemma4_kvs(&mut self, body: &str, events: &EventSink<'_>) -> Result<(), ()> {
        use super::sse::GenerationEvent;
        // Walk from scan_cursor, tracking `<|"|>` string state. On a top-level
        // `,` after `scan_cursor`, parse the kv span [scan_cursor..comma] and
        // emit. Advance scan_cursor past the comma.
        let bytes = body.as_bytes();
        let mut in_str = false;
        let mut i = self.scan_cursor;
        let kv_start = self.scan_cursor;
        let mut last_kv_start = kv_start;
        while i < bytes.len() {
            if !in_str && bytes[i..].starts_with(b"<|\"|>") {
                in_str = true;
                i += 5;
                continue;
            }
            if in_str && bytes[i..].starts_with(b"<|\"|>") {
                in_str = false;
                i += 5;
                continue;
            }
            if !in_str && bytes[i] == b',' {
                let kv = &body[last_kv_start..i];
                if let Some(json) = gemma4_kv_to_json(kv) {
                    let prefix = if self.kvs_emitted == 0 { "" } else { "," };
                    let frag = format!("{prefix}{json}");
                    if events
                        .blocking_send(GenerationEvent::ToolCallDelta {
                            index: self.index,
                            id: None,
                            call_type: None,
                            name: None,
                            arguments: Some(frag),
                        })
                        .is_err()
                    {
                        return Err(());
                    }
                    self.kvs_emitted += 1;
                    last_kv_start = i + 1;
                    self.scan_cursor = i + 1;
                }
                i += 1;
                continue;
            }
            // `}` at top level marks the args object's close. Stop scanning —
            // finalize will emit the trailing kv (if any) + `}`. Don't
            // speculatively emit on `}` because the body may still gain bytes
            // (sticky tool_close marker streamed separately).
            if !in_str && bytes[i] == b'}' {
                break;
            }
            i += 1;
        }
        Ok(())
    }

    /// Scan-and-emit closed Qwen 3.5/3.6 `<parameter=KEY>VAL</parameter>`
    /// blocks from `body[scan_cursor..]`. A block is "closed" when we observe
    /// `</parameter>` after its opening tag. Emits one delta per closed block.
    fn scan_emit_qwen35_kvs(&mut self, body: &str, events: &EventSink<'_>) -> Result<(), ()> {
        use super::sse::GenerationEvent;
        loop {
            // Locate the next `<parameter=` in body[scan_cursor..].
            let rest = &body[self.scan_cursor..];
            let Some(rel_open) = rest.find("<parameter=") else {
                break;
            };
            let p_open = self.scan_cursor + rel_open;
            let key_start = p_open + "<parameter=".len();
            let Some(rel_gt) = body[key_start..].find('>') else {
                break;
            };
            let key_end = key_start + rel_gt;
            let val_start = key_end + 1;
            let Some(rel_close) = body[val_start..].find("</parameter>") else {
                break;
            };
            let val_end = val_start + rel_close;
            let after_close = val_end + "</parameter>".len();
            let key = body[key_start..key_end].trim();
            let val_raw = body[val_start..val_end].trim();
            if key.is_empty() {
                // Malformed — leave scan_cursor where it is so finalize can
                // exercise the close-buffered loud-error branch under
                // policy.enforces_body_grammar(). Stop streaming this block.
                break;
            }
            let json_val: serde_json::Value = match serde_json::from_str(val_raw) {
                Ok(v) => v,
                Err(_) => serde_json::Value::String(val_raw.to_string()),
            };
            let key_json = serde_json::to_string(key).unwrap_or_else(|_| format!("\"{key}\""));
            let val_json = serde_json::to_string(&json_val).unwrap_or_else(|_| "null".to_string());
            let prefix = if self.kvs_emitted == 0 { "" } else { "," };
            let frag = format!("{prefix}{key_json}:{val_json}");
            if events
                .blocking_send(GenerationEvent::ToolCallDelta {
                    index: self.index,
                    id: None,
                    call_type: None,
                    name: None,
                    arguments: Some(frag),
                })
                .is_err()
            {
                return Err(());
            }
            self.kvs_emitted += 1;
            self.scan_cursor = after_close;
        }
        Ok(())
    }

    /// Emit the close-time tail. Two routes:
    ///
    ///   - **Streaming path was active** (`name_emitted == true`): re-parse
    ///     the full body to recover the last (un-streamed) kv plus the args
    ///     close. Emit the residual JSON tail and the closing `}` as one
    ///     final arguments delta. Increment `tc_index` and set `saw_tc`.
    ///     Returns `Ok(())`.
    ///
    ///     Wave 3.5 MED honesty note: this branch fires for any
    ///     well-formed body whose first `advance` call could extract
    ///     the function name from a prefix — INCLUDING single-fragment
    ///     bodies where the entire `call:NAME{...}` (Gemma 4) or
    ///     `<function=NAME>...</function>` (Qwen 3.5/3.6) arrived in
    ///     one fragment.  The audit at
    ///     `/tmp/cfa-cfa-20260427-adr005-wave3/codex-review-last.txt`
    ///     (divergence "W-B3 single-fragment fallback" severity MED)
    ///     correctly observed that no "single-fragment legacy
    ///     fallback" exists for well-formed bodies — `advance` always
    ///     emits chunk 1 (id+name) + chunk 2 (`{`) immediately on
    ///     well-formed input.  The incremental shape IS the canonical
    ///     OpenAI streaming contract; there is no client-visible
    ///     "two-chunk close-buffered" shape for well-formed
    ///     single-fragment bodies under Wave 3 W-B3 + later.
    ///
    ///   - **Streaming path never fired** (`name_emitted == false`): the
    ///     emitter declined the body (unknown family, OR name didn't appear
    ///     in any prefix). Delegate to `emit_streaming_tool_call_close` so
    ///     the close-buffered shape AND the policy-enforced loud-error
    ///     branches stay byte-for-byte identical to pre-W-B3 behaviour.
    ///     This branch is exercised by
    ///     `streaming_unknown_family_falls_back_to_legacy`.
    fn finalize(
        &mut self,
        body: String,
        registration: Option<&super::registry::ModelRegistration>,
        policy: ToolCallPolicy,
        tc_index: &mut usize,
        saw_tc: &mut bool,
        events: &EventSink<'_>,
    ) -> Result<(), ()> {
        use super::sse::GenerationEvent;

        if !self.name_emitted {
            // Fallback: streaming path never fired. Use the legacy close-
            // buffered emit so policy-enforced loud-error branches and the
            // single-chunk shape both stay intact.
            let parsed = registration.and_then(|r| super::registry::parse_tool_call_body(r, &body));
            return emit_streaming_tool_call_close(parsed, body, policy, tc_index, saw_tc, events);
        }

        // Streaming path was active. Re-parse the now-complete body and emit
        // the tail (last kv we couldn't stream because we couldn't
        // distinguish "final kv" from "next kv arriving later") + the
        // closing `}`.
        let parsed = registration.and_then(|r| super::registry::parse_tool_call_body(r, &body));
        let Some(pc) = parsed else {
            // Body failed parse despite streaming having extracted the name.
            // Under policy.enforces_body_grammar(), this means the grammar
            // engine produced bytes the per-family parser can't reassemble —
            // an unreachable-fallback regression. Promote to loud Error.
            // Under Auto (no grammar), preserve content fallback semantics
            // by closing the streaming JSON args we already emitted with `}`
            // (so the client's accumulator is at least valid JSON for the
            // partial it received) and then NOT emitting the residue as a
            // re-content delta — the partial args we streamed are the
            // semantically-faithful slice we managed to extract.
            if policy.enforces_body_grammar() {
                tracing::error!(
                    body = %body,
                    "tool_call_unreachable_fallback_required: body unparseable \
                     after streaming name extraction; per-family grammar bug"
                );
                let _ = events.blocking_send(GenerationEvent::Error(
                    "tool_call_unreachable_fallback_required".into(),
                ));
                return Err(());
            }
            // Auto-no-grammar: close the streaming JSON args we already
            // committed to so client accumulators land on valid JSON.
            if events
                .blocking_send(GenerationEvent::ToolCallDelta {
                    index: self.index,
                    id: None,
                    call_type: None,
                    name: None,
                    arguments: Some("}".into()),
                })
                .is_err()
            {
                return Err(());
            }
            *tc_index += 1;
            *saw_tc = true;
            return Ok(());
        };

        // Reconstruct the exact JSON args string emitted so far ( = `{` plus
        // each kv-comma-separated ) and compute the residual tail by
        // diffing against `pc.arguments_json`. This is robust to:
        //   - emitter scanned 0 kvs (whole args arrived in the close fragment)
        //   - emitter scanned all-but-last kv (typical streaming case)
        //   - emitter scanned all kvs (rare: comma after final kv would have
        //     to appear in body, which Gemma's template doesn't emit; Qwen
        //     trailing `</parameter>` followed by `</function>` does mean
        //     scan_cursor is past the last kv before finalize)
        let so_far = self.reconstruct_emitted_args_prefix(&pc.arguments_json);
        let tail = pc.arguments_json[so_far.len()..].to_string();
        if !tail.is_empty() {
            if events
                .blocking_send(GenerationEvent::ToolCallDelta {
                    index: self.index,
                    id: None,
                    call_type: None,
                    name: None,
                    arguments: Some(tail),
                })
                .is_err()
            {
                return Err(());
            }
        }
        *tc_index += 1;
        *saw_tc = true;
        Ok(())
    }

    /// Reconstruct the JSON-args string the streaming emitter has *already*
    /// sent to the client, so `finalize` can compute the residual tail by
    /// suffix-diff against the full `arguments_json` returned by
    /// `parse_tool_call_body`.
    ///
    /// Strategy: walk `full_args_json` (which is well-formed `{...}`) and
    /// take the longest prefix that contains exactly `kvs_emitted` top-level
    /// kvs. The streaming emitter always emits `{`, then for kv #1 just the
    /// raw kv JSON, then for kv #2..N a leading `,`. Therefore the prefix
    /// we already emitted ends RIGHT BEFORE the start of kv #(kvs_emitted+1)
    /// — i.e. before the comma preceding it (or before the `}` if all kvs
    /// were streamed).
    fn reconstruct_emitted_args_prefix<'a>(&self, full_args_json: &'a str) -> &'a str {
        // Walk the JSON object counting kv-pairs at depth 1. We can use a
        // simple state machine: track `{}` depth and `"` string state, count
        // commas at depth 1 (each comma = boundary between two kvs).
        let bytes = full_args_json.as_bytes();
        let mut depth: i32 = 0;
        let mut in_str = false;
        let mut esc = false;
        let mut commas_at_depth_1 = 0usize;
        // Number of kvs in the prefix we've sent = self.kvs_emitted.
        // Number of commas in that prefix = max(0, kvs_emitted - 1) + (1 if
        // kvs_emitted > 0 we've emitted up to and including kv #N, NOT
        // beyond it). So we want the longest prefix ending RIGHT BEFORE
        // `,` #(kvs_emitted) or, if we've emitted 0 kvs, right after the
        // opening `{`.
        if self.kvs_emitted == 0 {
            // Emitted only `{`. Prefix is `{`.
            // Find the first `{` (well-formed JSON starts with it).
            for (i, &b) in bytes.iter().enumerate() {
                if b == b'{' {
                    return &full_args_json[..=i];
                }
            }
            return "";
        }
        // We need to find the position of the (kvs_emitted)th comma at
        // depth 1, OR the closing `}` at depth 1 if no further comma exists
        // — and return the prefix ending just before it.
        for (i, &b) in bytes.iter().enumerate() {
            if in_str {
                if esc {
                    esc = false;
                } else if b == b'\\' {
                    esc = true;
                } else if b == b'"' {
                    in_str = false;
                }
                continue;
            }
            match b {
                b'"' => in_str = true,
                b'{' | b'[' => depth += 1,
                b'}' | b']' => {
                    depth -= 1;
                    if depth == 0 && commas_at_depth_1 + 1 == self.kvs_emitted {
                        // No further comma — we've streamed every kv. The
                        // prefix is everything up to (not including) `}`.
                        return &full_args_json[..i];
                    }
                }
                b',' => {
                    if depth == 1 {
                        commas_at_depth_1 += 1;
                        if commas_at_depth_1 == self.kvs_emitted {
                            // The Nth comma at depth 1 separates kv #N from
                            // kv #(N+1). The prefix we've emitted ends
                            // RIGHT BEFORE this comma (since kv #(N+1) hasn't
                            // been streamed; finalize will emit `,kv#(N+1)`
                            // — so the residue must include the comma).
                            return &full_args_json[..i];
                        }
                    }
                }
                _ => {}
            }
        }
        // Defensive: malformed input — return full string so tail is empty.
        full_args_json
    }
}

/// Extract the function name from a Gemma 4 body prefix `call:NAME{`. Returns
/// `Some((name, header_end))` where `header_end` is the byte offset of the
/// `{` (so `body[header_end+1..]` is the kv-list region the streaming
/// scanner walks). Returns `None` if the `{` hasn't arrived yet OR the
/// extracted name fails OpenAI-spec validity (iter-219b: rejects
/// special-token-polluted names that the splitter couldn't trim).
fn extract_gemma4_name_prefix(body: &str) -> Option<(String, usize)> {
    let trimmed_offset = body.len() - body.trim_start().len();
    let after_ws = &body[trimmed_offset..];
    let after_call = after_ws.strip_prefix("call:")?;
    let brace_rel = after_call.find('{')?;
    let name = after_call[..brace_rel].trim().to_string();
    if !super::registry::is_valid_tool_name(&name) {
        return None;
    }
    let absolute_brace = trimmed_offset + "call:".len() + brace_rel;
    Some((name, absolute_brace + 1))
}

/// Extract the function name from a Qwen 3.5/3.6 body prefix
/// `<function=NAME>`. Returns `Some((name, header_end))` where `header_end`
/// is the byte offset just past `>` (so the streaming scanner walks
/// `body[header_end..]` for `<parameter>` blocks). Returns `None` if the
/// closing `>` hasn't arrived yet OR the extracted name fails OpenAI-spec
/// validity.
fn extract_qwen35_name_prefix(body: &str) -> Option<(String, usize)> {
    let trimmed_offset = body.len() - body.trim_start().len();
    let after_ws = &body[trimmed_offset..];
    let after_open = after_ws.strip_prefix("<function=")?;
    let gt_rel = after_open.find('>')?;
    let name = after_open[..gt_rel].trim().to_string();
    if !super::registry::is_valid_tool_name(&name) {
        return None;
    }
    let absolute_gt = trimmed_offset + "<function=".len() + gt_rel;
    Some((name, absolute_gt + 1))
}

/// Convert one Gemma 4 kv span `key:<jsonval>` into a JSON `"key":<json>`
/// fragment. Mirrors the value-coercion logic in `parse_gemma4_tool_call`
/// at registry.rs:737-768. Returns `None` on malformed kv (caller leaves
/// scan_cursor untouched so finalize falls into the close-buffered path).
fn gemma4_kv_to_json(kv: &str) -> Option<String> {
    let (k, v) = kv.split_once(':')?;
    let key = k.trim();
    if key.is_empty() {
        return None;
    }
    let v = v.trim();
    let json_val = if let Some(stripped) = v
        .strip_prefix("<|\"|>")
        .and_then(|s| s.strip_suffix("<|\"|>"))
    {
        serde_json::Value::String(stripped.to_string())
    } else if let Ok(num) = v.parse::<i64>() {
        serde_json::Value::from(num)
    } else if let Ok(num) = v.parse::<f64>() {
        serde_json::Value::from(num)
    } else if v == "true" {
        serde_json::Value::Bool(true)
    } else if v == "false" {
        serde_json::Value::Bool(false)
    } else if v == "null" {
        serde_json::Value::Null
    } else {
        serde_json::Value::String(v.to_string())
    };
    let key_json = serde_json::to_string(key).ok()?;
    let val_json = serde_json::to_string(&json_val).ok()?;
    Some(format!("{key_json}:{val_json}"))
}

/// Dispatch the close-time tool-call body after `ToolCallClose` fires in the
/// streaming `route_content` closure.
///
/// Wave 3 W-A3 — T2.4 partial removal (Constrained body-parse-failure case).
/// Wave 3 W-B2 — T2.4 final closure: `AutoLazyGrammar` joins `Constrained`
/// in the loud-error branch.
///
/// ## Behaviour matrix
///
/// ```text
///                       │ Constrained / AutoLazyGrammar   │ Auto (no grammar)
/// ─────────────────────┼─────────────────────────────────┼────────────────────────────────
/// parse OK              │ emit ToolCallDelta ×2           │ emit ToolCallDelta ×2
/// parse FAILURE         │ emit GenerationEvent::Error     │ emit delta.content fallback
///                       │   "tool_call_unreachable_       │   (tracing::warn)
///                       │    fallback_required"           │
///                       │ + tracing::error! (grammar bug) │
///                       │ → Err(())                       │ → Ok(())  [or Err if send fails]
/// ```
///
/// **Why Auto (no grammar) preserves the content fallback**: when the
/// request is `tool_choice=auto` AND no grammar is active (no tools[]
/// declared, OR an unregistered model family) there is no enforcement on
/// body shape. A model can legitimately emit malformed / partial
/// tool-call syntax and the caller should still see the raw bytes rather
/// than losing them silently. The content fallback is the defined
/// behaviour for this unconstrained branch.
///
/// **Why Constrained AND AutoLazyGrammar both error loudly**: under
/// `Constrained` the wave-2.7 W-η da545d5 eager grammar constrains every
/// token from byte 0. Under `AutoLazyGrammar` (wave 3 W-B2) the same
/// per-model body grammar is active from `ToolCallOpen` onwards via the
/// `awaiting_trigger` gate flip. In both cases the grammar physically
/// constrains the body bytes, so a body-parse failure means the grammar
/// engine produced structurally invalid output — a server-side
/// regression, not a model quality issue. Surfacing it as a loud error
/// (rather than a silent content fallback) makes the regression
/// immediately visible to operators.
///
/// The unified gate is `policy.enforces_body_grammar()` — see
/// `ToolCallPolicy::enforces_body_grammar` for the single source of
/// truth.
///
/// Extracted from the `route_content` closure so audit-driver tests can
/// exercise this exact code path directly (same extraction pattern as
/// `finalize_streaming_tool_state` for HIGH-1).
///
/// Wedge-3 / iter-216: surfaced as `pub(super)` so the Qwen3.5/3.6
/// streaming arm in `engine_qwen35::generate_stream_qwen35_once` can
/// reuse the same close-buffered tool-call dispatch the Gemma path
/// uses, keeping the body-parse-failure semantics + ToolCallDelta
/// shape byte-identical across model families.
pub(super) fn emit_streaming_tool_call_close(
    parsed: Option<super::registry::ParsedToolCall>,
    body_dump: String,
    policy: ToolCallPolicy,
    tc_index: &mut usize,
    saw_tc: &mut bool,
    events: &EventSink<'_>,
) -> Result<(), ()> {
    use super::sse::{DeltaKind, GenerationEvent};

    match parsed {
        Some(pc) => {
            // First chunk: id + type + name. The id is a synthesized opaque
            // identifier; clients echo it in their `tool_call_id` follow-up
            // message. Format mirrors OpenAI's `call_<24hex>` shape.
            let id = format!(
                "call_hf2q_{:016x}",
                std::time::SystemTime::now()
                    .duration_since(std::time::UNIX_EPOCH)
                    .map(|d| d.as_nanos() as u64)
                    .unwrap_or(0)
                    ^ (*tc_index as u64).wrapping_mul(0x9e3779b97f4a7c15)
            );
            if events
                .blocking_send(GenerationEvent::ToolCallDelta {
                    index: *tc_index,
                    id: Some(id),
                    call_type: Some("function".into()),
                    name: Some(pc.name),
                    arguments: None,
                })
                .is_err()
            {
                return Err(());
            }
            // Second chunk: full arguments JSON string. OpenAI clients
            // accumulate `function.arguments` deltas; one chunk is spec-valid.
            if events
                .blocking_send(GenerationEvent::ToolCallDelta {
                    index: *tc_index,
                    id: None,
                    call_type: None,
                    name: None,
                    arguments: Some(pc.arguments_json),
                })
                .is_err()
            {
                return Err(());
            }
            *tc_index += 1;
            *saw_tc = true;
            Ok(())
        }
        None => {
            // Wave 3 W-A3 + W-B2 — T2.4 final closure on registered families.
            //
            // Both Constrained AND AutoLazyGrammar carry an active grammar
            // that physically constrains the body — Constrained from byte 0
            // (eager), AutoLazyGrammar from `ToolCallOpen` onwards (lazy).
            // A parse failure under either policy means the grammar engine
            // produced structurally invalid output. Promote to a loud
            // structured error so the regression surfaces immediately
            // rather than being silently swallowed by the content fallback.
            //
            // Auto (no grammar): legitimate parse-failure path. The model
            // emitted malformed syntax with no grammar to constrain it;
            // re-emit as content so the caller sees what the model intended.
            if policy.enforces_body_grammar() {
                let policy_label = match policy {
                    ToolCallPolicy::Constrained => "constrained (required/function)",
                    ToolCallPolicy::AutoLazyGrammar => "auto-lazy-grammar (wave-3 W-B2)",
                    ToolCallPolicy::Auto => unreachable!("enforces_body_grammar gate"),
                };
                tracing::error!(
                    body = %body_dump,
                    policy = policy_label,
                    "tool_call_unreachable_fallback_required: tool-call body \
                     unparseable under {} policy; the per-model body grammar \
                     should have prevented this — grammar engine bug",
                    policy_label
                );
                let _ = events.blocking_send(GenerationEvent::Error(
                    "tool_call_unreachable_fallback_required".into(),
                ));
                return Err(());
            }
            // Auto (no grammar) path: preserve the pre-wave-2.5 content
            // fallback. (See function doc above for the rationale.)
            //
            // iter-219b (2026-05-01): scrub registered in-call special-token
            // markers from the body before emitting. Without this scrub,
            // a polluted body (e.g. with `<|tool_response>` mid-call —
            // see `iter219b_reproducer_tool_response_inside_call`) would
            // leak the special-token literal into `delta.content` via this
            // fallback path, re-introducing the iter-217-class leak that
            // `assert_no_leaked_special_tokens` is supposed to catch.
            let scrubbed = super::registry::scrub_special_tokens(&body_dump);
            tracing::warn!(
                body = %body_dump,
                scrubbed = %scrubbed,
                "tool-call body unparseable; emitting as content fallback \
                 (tool_choice=auto with no active grammar — no enforcement \
                 on body shape; either tools[] empty or unregistered family). \
                 Special-token markers scrubbed from body before emit."
            );
            if events
                .blocking_send(GenerationEvent::Delta {
                    kind: DeltaKind::Content,
                    text: scrubbed,
                })
                .is_err()
            {
                return Err(());
            }
            Ok(())
        }
    }
}

/// Replay a cached `GenerationResult` as a sequence of SSE events.
///
/// Wave 3 W-A2 — closes the asymmetry documented at the iter-96 streaming
/// store-only callsite.  Pre-W-A2 the streaming path stored to
/// `PromptCache` on every successful completion but never consulted the
/// cache on input, so most clients (which use streaming) never benefited
/// from cache hits.  This helper lets `generate_stream_once` short-circuit
/// when `PromptCache::lookup` returns `Some(...)`: emit the cached text as
/// SSE deltas (re-classified through the same Reasoning + ToolCall splitter
/// pipeline the live decode uses), then emit `Done` with zero timings and
/// `cached_prompt_tokens = Some(prompt_len)` so the client surfaces
/// `usage.prompt_tokens_details.cached_tokens` exactly like a non-streaming
/// hit.
///
/// # Cache-shape decision (deliberate, surfaced in commit body)
///
/// `PromptCache` stores text only — no token sequence — so the replay
/// CANNOT preserve original token boundaries (one delta per cached token).
/// Two designs were considered:
///
/// 1. **Single big content delta.** Simple. Loses tool-call shape if the
///    cached response was a tool call: the open/close markers would arrive
///    inside `delta.content` instead of producing structured
///    `delta.tool_calls[*]` events.  Spec-violating for tool-call replays.
///
/// 2. **Re-route through the live splitter pipeline (chosen).** Build a
///    fresh `ReasoningSplitter` + `ToolCallSplitter` from the model
///    registration (same factory the live decode uses), feed the cached
///    `text` in once, and dispatch the resulting events through the same
///    `route_content` / `emit_streaming_tool_call_close` helpers.  When
///    the cached text contains tool-call markers, the splitter emits
///    structured `ToolCallDelta` events identical to a fresh decode.
///    When the text is plain content, the splitter emits a single
///    `Content` delta.  When `reasoning_text` is `Some(...)` (cached from
///    a non-streaming completion that already split reasoning out), it is
///    emitted first as a `Reasoning` delta so the SSE response shape
///    matches the original.
///
/// Per-cached-token replay (preserve TTFT-like incremental UX) requires
/// extending `PromptCache` to store the per-token Delta sequence rather
/// than the assembled text.  That is a real shape extension; documented as
/// a follow-up rather than shoehorned into this iter.
///
/// # Why pass `tool_call_policy`
///
/// Tool-call body parse failures branch on policy (Constrained → loud
/// `GenerationEvent::Error`; Auto → silent content fallback).  The cache
/// key includes `tool_call_policy`, so a hit guarantees the policy
/// matches the original request — but `emit_streaming_tool_call_close`
/// still requires it as an argument to make the branch explicit.
///
/// # Why `grammar_runtime: None`
///
/// `route_content`'s `ToolCallOpen` branch flips
/// `runtime.is_awaiting_trigger()` so subsequent decode-loop mask calls
/// fire.  In replay there is NO decode loop — the runtime is irrelevant.
/// Pass `None` so the branch is a no-op (matches the no-grammar live
/// path).  This is sound because the trigger is purely a live-decode
/// gating mechanism, not part of the SSE event shape.
///
/// Returns `Ok(())` if the full cached response (Reasoning? + content
/// events + Done) was emitted; `Err(())` if any send failed (client
/// disconnected mid-replay) — caller bumps the cancellation counter and
/// returns, mirroring the live-decode disconnect path.
///
/// # End-of-stream splitter drain (Wave 3.5 HIGH-2)
///
/// Both `ReasoningSplitter` and `ToolCallSplitter` hold back a sliding
/// tail (`tail_buf`) up to `tail_cap` bytes long in case the next
/// fragment continues a marker boundary.  Pre-Wave-3.5 the replay fed
/// `cached.text` once and emitted Done — never calling `finish()` on
/// either splitter — so the held-back tail bytes were silently dropped.
/// This caused truncated content on cache hits whose tails happened to
/// look like partial markers.
///
/// The drain order mirrors the live-decode path
/// (engine.rs:3691-3757): `reasoning_splitter.finish()` first, routing
/// any Content tail through `tool_splitter`; then
/// `tool_splitter.finish()` to emit the final residual.  Unlike the
/// live path, replay does NOT promote ToolCallText residuals to
/// structured Errors — a cached entry was already validated when
/// stored, so any residual tail is plain content not a mid-decode
/// truncation.  Audit citation:
/// `/tmp/cfa-cfa-20260427-adr005-wave3/codex-review-last.txt`
/// divergence "W-A2 streaming cache replay" severity HIGH.
fn replay_cached_streaming_response(
    cached: &GenerationResult,
    registration: Option<&super::registry::ModelRegistration>,
    tool_call_policy: ToolCallPolicy,
    events: &EventSink<'_>,
) -> Result<(), ()> {
    // W-A2.3 default callers (tests, helpers without fragment access) get
    // the splitter-rerun replay path — fragments=None preserves the Wave-3.5
    // HIGH-2 tail_buf drain.  Streaming origin (`generate_stream_once` cache
    // hit) drives `replay_cached_streaming_response_with_fragments` directly
    // with `Some(frags)` to bypass the splitter pipeline entirely and emit
    // byte-identical event-stream framing.
    replay_cached_streaming_response_with_fragments(
        cached,
        registration,
        tool_call_policy,
        events,
        None,
    )
}

/// W-A2.3 fragments-aware streaming-cache replay.
///
/// Branches on `cached_fragments`:
///
/// - `Some(frags)` — **fragments-replay branch**.  Emit each
///   `CachedFragment` directly as the matching `GenerationEvent`, then
///   emit the terminal `Done`.  Skips the ReasoningSplitter +
///   ToolCallSplitter pipeline, the splitter feeds, AND the
///   end-of-stream `tail_buf` drain (Wave-3.5 HIGH-2 fix at
///   engine.rs:4332).  The drain is unnecessary on this branch
///   because the splitters never run — there is no held-back tail to
///   drop.  Captured at streaming origin, so per-token boundaries are
///   preserved byte-for-byte (the W-A2 closure UX win).
///
/// - `None` — **legacy splitter-rerun branch** (preserves the
///   Wave-3.5 HIGH-2 splitter drain).  Builds fresh
///   ReasoningSplitter + ToolCallSplitter from the registration,
///   feeds `cached.text` through them, and drains both at end of
///   stream.  This path is for non-streaming-origin entries (no
///   per-token trace exists) and for any test/legacy caller of
///   `replay_cached_streaming_response`.
///
/// The fragments branch is the byte-identical-event-stream contract
/// — Worker AA design §6 falsifiable closure.
fn replay_cached_streaming_response_with_fragments(
    cached: &GenerationResult,
    registration: Option<&super::registry::ModelRegistration>,
    tool_call_policy: ToolCallPolicy,
    events: &EventSink<'_>,
    cached_fragments: Option<&Vec<CachedFragment>>,
) -> Result<(), ()> {
    use super::sse::{DeltaKind, GenerationEvent, StreamStats};

    // ── 0. Fragments-replay branch (W-A2.3) ─────────────────────────────
    //
    // Streaming-origin entry: emit each captured fragment directly as the
    // matching GenerationEvent.  No splitter pipeline, no tail_buf drain —
    // the W-A2.2 capture mirrors EVERY emitted Delta / ToolCallDelta into
    // the vec, so the splitter run that originally produced these emits
    // does not need to be re-run.
    //
    // Critical Chesterton-fence note: this branch MUST emit the SAME
    // terminal `Done` event the splitter-rerun branch emits below
    // (cached_prompt_tokens populated, timings zeroed).  If the Done
    // shape diverges, the byte-identity contract from Worker AA §6
    // breaks.  The Done emit is shared across both branches by falling
    // through after the fragments emit.
    if let Some(frags) = cached_fragments {
        // saw_tool_call: a streaming-origin capture that produced any
        // ToolCallDelta means the live decode reached at least one
        // ToolCallClose, which is the same trigger
        // `generate_stream_once`'s `saw_tool_call` flag uses.  Mirror
        // the live `Done.finish_reason = "tool_calls"` override here so
        // the cache replay's terminal chunk matches the original
        // request's terminal chunk.
        let mut saw_tool_call = false;
        for frag in frags {
            match frag {
                CachedFragment::Content(text) => {
                    if !text.is_empty()
                        && events
                            .blocking_send(GenerationEvent::Delta {
                                kind: DeltaKind::Content,
                                text: text.clone(),
                            })
                            .is_err()
                    {
                        return Err(());
                    }
                }
                CachedFragment::Reasoning(text) => {
                    if !text.is_empty()
                        && events
                            .blocking_send(GenerationEvent::Delta {
                                kind: DeltaKind::Reasoning,
                                text: text.clone(),
                            })
                            .is_err()
                    {
                        return Err(());
                    }
                }
                CachedFragment::ToolCallDelta {
                    index,
                    id,
                    call_type,
                    name,
                    arguments,
                } => {
                    saw_tool_call = true;
                    if events
                        .blocking_send(GenerationEvent::ToolCallDelta {
                            index: *index,
                            id: id.clone(),
                            call_type: call_type.clone(),
                            name: name.clone(),
                            arguments: arguments.clone(),
                        })
                        .is_err()
                    {
                        return Err(());
                    }
                }
            }
        }
        // Emit Done — same shape as the splitter-rerun branch below.
        // Tool-call replay overrides finish_reason per OpenAI spec.
        let stats = StreamStats {
            prefill_time_secs: Some(0.0),
            decode_time_secs: Some(0.0),
            total_time_secs: Some(0.0),
            time_to_first_token_ms: Some(0.0),
            prefill_tokens_per_sec: None,
            decode_tokens_per_sec: None,
            gpu_sync_count: None,
            gpu_dispatch_count: None,
            cached_prompt_tokens: Some(cached.cached_tokens),
            reasoning_tokens: cached.reasoning_tokens,
        };
        if events
            .blocking_send(GenerationEvent::Done {
                finish_reason: if saw_tool_call {
                    "tool_calls"
                } else {
                    cached.finish_reason
                },
                prompt_tokens: cached.prompt_tokens,
                completion_tokens: cached.completion_tokens,
                stats,
            })
            .is_err()
        {
            return Err(());
        }
        return Ok(());
    }

    // ── 1. Reasoning replay ─────────────────────────────────────────────
    //
    // Non-streaming-origin cache entries store reasoning_text separately
    // (split out of the assembled text via `split_full_output` in
    // `generate_once_with_soft_tokens`).  Streaming-origin entries store
    // reasoning_text == None because the live splitter routed reasoning
    // fragments into Reasoning deltas as decoded; the assembled text
    // contains the full pre-split stream.  Either way: emit the
    // explicit reasoning_text first (if any), then route the text through
    // the splitter to handle the streaming-origin embedded-marker case.
    if let Some(reasoning) = cached.reasoning_text.as_deref() {
        if !reasoning.is_empty()
            && events
                .blocking_send(GenerationEvent::Delta {
                    kind: DeltaKind::Reasoning,
                    text: reasoning.to_string(),
                })
                .is_err()
        {
            return Err(());
        }
    }

    // ── 2. Content / tool-call replay ───────────────────────────────────
    //
    // Build fresh splitters mirroring the `generate_stream_once` setup
    // (lines below the lookup).  The ReasoningSplitter handles any
    // embedded reasoning markers (streaming-origin entries).  The
    // ToolCallSplitter handles embedded tool-call markers regardless of
    // origin (neither streaming nor non-streaming strips them).
    //
    // iter-230 B: forced_open = false here BY DESIGN — this replays a
    // CACHED GenerationResult whose text was already split at capture
    // time (the original request's seed applied then); the cached
    // content field starts OUTSIDE any reasoning span.
    let mut reasoning_splitter =
        registration.and_then(|r| super::registry::make_reasoning_splitter(r, false));
    let mut tool_splitter =
        registration.and_then(|r| super::registry::ToolCallSplitter::from_registration(r));

    let mut tool_call_body: String = String::new();
    let mut tool_call_index: usize = 0;
    let mut saw_tool_call: bool = false;

    // Replay-side fragment routing.  Mirrors `route_content` minus
    // grammar plumbing (no decode loop ⇒ no runtime to trigger).  Inline
    // here rather than reusing the closure because the closure captures
    // generate_stream_once-local state we don't have in this free fn —
    // duplicating the ~30-line dispatch is cheaper than threading a
    // borrow web across a closure parameter list.  Architecturally this
    // is the same pattern as `emit_streaming_tool_call_close` (extracted
    // for the close-branch) and `finalize_streaming_tool_state` (extracted
    // for the end-of-stream drain).
    let route_replay_fragment = |tcs: &mut Option<super::registry::ToolCallSplitter>,
                                 body: &mut String,
                                 tc_index: &mut usize,
                                 saw_tc: &mut bool,
                                 events: &EventSink<'_>,
                                 text: &str|
     -> Result<(), ()> {
        if text.is_empty() {
            return Ok(());
        }
        let Some(tcs) = tcs.as_mut() else {
            if events
                .blocking_send(GenerationEvent::Delta {
                    kind: DeltaKind::Content,
                    text: text.to_string(),
                })
                .is_err()
            {
                return Err(());
            }
            return Ok(());
        };
        for ev in tcs.feed(text) {
            match ev {
                super::registry::ToolCallEvent::Content(t) => {
                    if !t.is_empty()
                        && events
                            .blocking_send(GenerationEvent::Delta {
                                kind: DeltaKind::Content,
                                text: t,
                            })
                            .is_err()
                    {
                        return Err(());
                    }
                }
                super::registry::ToolCallEvent::ToolCallOpen => {
                    body.clear();
                    // Replay: no grammar_runtime to trigger.
                }
                super::registry::ToolCallEvent::ToolCallText(t) => {
                    body.push_str(&t);
                }
                super::registry::ToolCallEvent::ToolCallClose => {
                    let parsed =
                        registration.and_then(|r| super::registry::parse_tool_call_body(r, body));
                    let body_dump = std::mem::take(body);
                    emit_streaming_tool_call_close(
                        parsed,
                        body_dump,
                        tool_call_policy,
                        tc_index,
                        saw_tc,
                        events,
                    )?;
                }
            }
        }
        Ok(())
    };

    // Feed the cached text through the reasoning splitter first, then
    // route Content-classified spans through the tool-call splitter.
    // Mirrors `emit_fragment` in generate_stream_once.
    if let Some(rs) = reasoning_splitter.as_mut() {
        for (slot, text) in rs.feed(&cached.text) {
            match slot {
                super::registry::SplitSlot::Reasoning => {
                    if !text.is_empty()
                        && events
                            .blocking_send(GenerationEvent::Delta {
                                kind: DeltaKind::Reasoning,
                                text,
                            })
                            .is_err()
                    {
                        return Err(());
                    }
                }
                super::registry::SplitSlot::Content => {
                    route_replay_fragment(
                        &mut tool_splitter,
                        &mut tool_call_body,
                        &mut tool_call_index,
                        &mut saw_tool_call,
                        events,
                        &text,
                    )?;
                }
            }
        }
    } else {
        route_replay_fragment(
            &mut tool_splitter,
            &mut tool_call_body,
            &mut tool_call_index,
            &mut saw_tool_call,
            events,
            &cached.text,
        )?;
    }

    // ── 2b. End-of-stream splitter drain ────────────────────────────────
    //
    // Wave 3.5 HIGH-2: live decode drains BOTH the ReasoningSplitter and
    // the ToolCallSplitter at end-of-stream because each holds back a
    // tail (`tail_buf` of size up to `tail_cap` bytes) in case the next
    // fragment continues a marker boundary.  Pre-Wave-3.5 replay fed
    // `cached.text` through `feed()` ONCE then jumped straight to Done,
    // never calling `finish()` on either splitter — so the held-back
    // tail bytes (typically a few characters) were silently dropped.
    //
    // Concrete failure mode the audit caught
    // (/tmp/cfa-cfa-20260427-adr005-wave3/codex-review-last.txt
    // divergence "W-A2 streaming cache replay" severity HIGH): a cached
    // plain-text response shorter than `tail_cap` bytes (or whose tail
    // looks like a partial marker prefix) had its terminal characters
    // truncated.  A cached response with a tool-call marker followed by
    // postscript content had the postscript truncated.
    //
    // Mirrors the live-decode drain at engine.rs:3691-3757
    // (reasoning_splitter.finish → tool_splitter Content route →
    // tool_splitter.finish via finalize_streaming_tool_state).  The
    // replay equivalent uses `route_replay_fragment` (no grammar
    // runtime) and inlines the tool_splitter.finish() drain because we
    // don't enforce mid-call truncation Errors on a cache hit (the
    // policy-loud-error contract is for live decode; cached entries
    // were already validated when stored).
    if let Some(rs) = reasoning_splitter.as_mut() {
        if let Some((slot, tail)) = rs.finish() {
            match slot {
                super::registry::SplitSlot::Reasoning => {
                    if !tail.is_empty()
                        && events
                            .blocking_send(GenerationEvent::Delta {
                                kind: DeltaKind::Reasoning,
                                text: tail,
                            })
                            .is_err()
                    {
                        return Err(());
                    }
                }
                super::registry::SplitSlot::Content => {
                    // Route through the tool-call splitter so a Content
                    // tail straddling a marker boundary still classifies
                    // correctly (mirrors live drain at engine.rs:3710-3727).
                    route_replay_fragment(
                        &mut tool_splitter,
                        &mut tool_call_body,
                        &mut tool_call_index,
                        &mut saw_tool_call,
                        events,
                        &tail,
                    )?;
                }
            }
        }
    }
    // tool_splitter.finish() drain: route any held-back tail to the
    // appropriate slot.  Unlike the live path (which fires a structured
    // Error on mid-call truncation under Constrained/AutoLazyGrammar),
    // replay always treats the tail as plain content for the same
    // reason: the cache stored a verified-complete response, so any
    // residual tail is by definition not a mid-decode truncation.
    if let Some(tcs) = tool_splitter.as_mut() {
        if let Some(ev) = tcs.finish() {
            match ev {
                super::registry::ToolCallEvent::Content(t) => {
                    if !t.is_empty()
                        && events
                            .blocking_send(GenerationEvent::Delta {
                                kind: DeltaKind::Content,
                                text: t,
                            })
                            .is_err()
                    {
                        return Err(());
                    }
                }
                super::registry::ToolCallEvent::ToolCallText(t) => {
                    // Cached entry ended mid-tool-call (open marker
                    // observed but no close marker reached).  Re-emit
                    // as Content with the open marker re-prepended for
                    // diagnostic clarity — same shape as the live
                    // Auto-no-grammar drain at engine.rs:2024-2035.
                    let prefix = registration.and_then(|r| r.tool_open).unwrap_or("");
                    let fallback = format!("{prefix}{t}");
                    if !fallback.is_empty()
                        && events
                            .blocking_send(GenerationEvent::Delta {
                                kind: DeltaKind::Content,
                                text: fallback,
                            })
                            .is_err()
                    {
                        return Err(());
                    }
                }
                super::registry::ToolCallEvent::ToolCallOpen
                | super::registry::ToolCallEvent::ToolCallClose => {
                    // unreachable — finish() never emits Open/Close.
                }
            }
        }
    }

    // ── 3. Done event ───────────────────────────────────────────────────
    //
    // `cached_prompt_tokens = Some(prompt_len)` so the SSE final-chunk
    // usage surfaces `prompt_tokens_details.cached_tokens` identically to
    // the non-streaming hit path (engine.rs:752-754).  Zero timings
    // because prefill+decode were skipped — same convention as the
    // non-streaming GenerationResult on hit.
    let stats = StreamStats {
        prefill_time_secs: Some(0.0),
        decode_time_secs: Some(0.0),
        total_time_secs: Some(0.0),
        time_to_first_token_ms: Some(0.0),
        prefill_tokens_per_sec: None,
        decode_tokens_per_sec: None,
        gpu_sync_count: None,
        gpu_dispatch_count: None,
        cached_prompt_tokens: Some(cached.cached_tokens),
        reasoning_tokens: cached.reasoning_tokens,
    };

    if events
        .blocking_send(GenerationEvent::Done {
            // Tool-call replays must override finish_reason so clients see
            // `tool_calls` (OpenAI spec). The splitter sets `saw_tool_call`
            // when a ToolCallClose fires.
            finish_reason: if saw_tool_call {
                "tool_calls"
            } else {
                cached.finish_reason
            },
            prompt_tokens: cached.prompt_tokens,
            completion_tokens: cached.completion_tokens,
            stats,
        })
        .is_err()
    {
        return Err(());
    }
    Ok(())
}

/// Streaming variant of `generate_once`. Sends `GenerationEvent::Delta` per
/// decoded token, followed by a terminating `Done` (with finish_reason +
/// usage) or `Error`. If the `events` receiver is dropped (SSE client
/// disconnect, Decision #18), the next `blocking_send` returns Err and the
/// loop exits early — no more events are sent, the queue slot is freed.
fn generate_stream_once(
    loaded: &mut GemmaLoadedModel,
    prompt_tokens: &[u32],
    soft_tokens: &[SoftTokenInjection<'_>],
    params: &SamplingParams,
    events: &tokio::sync::mpsc::Sender<super::sse::GenerationEvent>,
    registration: Option<&super::registry::ModelRegistration>,
    cancellation_counter: Option<&std::sync::atomic::AtomicU64>,
) {
    use super::sse::{DeltaKind, GenerationEvent, StreamStats};

    // W-A2.2: streaming origin captures the per-emit sequence into a
    // sibling `Vec<CachedFragment>`.  The capture is wrapped in a `RefCell`
    // so the `EventSink` (which all helpers borrow shared) can borrow_mut
    // through `EventSink::blocking_send` without conflicting with the
    // closures that capture `&events`.  The vec is consumed at end-of-
    // stream and passed to `prompt_cache.store_with_fragments` —
    // see store callsite below the decode loop.
    let captured_fragments: std::cell::RefCell<Vec<CachedFragment>> =
        std::cell::RefCell::new(Vec::new());
    let sink = EventSink::with_capture(events, &captured_fragments);
    // Helpers + closures borrow `events` as `&EventSink<'_>`; alias for
    // body uniformity (the original `events` ident shadowed below).
    let events = &sink;

    // Helper: send an event; if the receiver is gone, bump the
    // cancellation counter (→ hf2q_sse_cancellations in /metrics) and bail.
    macro_rules! send {
        ($ev:expr) => {
            if events.blocking_send($ev).is_err() {
                tracing::info!("SSE stream dropped by client; aborting decode");
                if let Some(c) = cancellation_counter {
                    c.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
                }
                return;
            }
        };
    }

    if prompt_tokens.is_empty() {
        send!(GenerationEvent::Error(
            "generate_stream_once: empty prompt_tokens".into()
        ));
        return;
    }
    let prompt_len = prompt_tokens.len();
    let max_tokens = params.max_tokens.max(1);

    // ── Prompt cache fast-path (Wave 3 W-A2) ──────────────────────────────
    //
    // Mirrors the non-streaming preroll lookup at engine.rs:1419 (iter-96
    // / wave-2.5 B5).  Pre-W-A2 the streaming path stored to PromptCache on
    // every successful completion (`store` call below the decode loop) but
    // never consulted it on input — most clients use streaming, so cache
    // hits were essentially impossible on the production path.  W-A2
    // closes the documented gap (former engine.rs:2109 "iter-97 follow-up"
    // comment).
    //
    // Eligibility: `PromptCache::lookup` self-gates on temperature/top_k/
    // top_p/repetition_penalty/seed (sampling-mode bypass), prompt-token
    // equality, AND PromptCacheKey full-inventory equality (wave-2.5 B5
    // expansion: max_tokens, stop_strings, logit_bias, grammar,
    // grammar_kind, frequency/presence/min_p penalties, tool_call_policy,
    // logprobs/top_logprobs, parallel_tool_calls).  See PromptCacheKey
    // doc at engine.rs:489-535 for the full inventory.
    //
    // On hit, the replay path emits the cached event sequence:
    //
    //   - **Streaming-origin entry** (W-A2.2 captured fragments): emit
    //     each `CachedFragment` directly as the matching
    //     `GenerationEvent`, byte-identical to the live event stream.
    //     This is the W-A2.3 fragments-replay branch — preserves
    //     per-token boundaries (perceived TTFT/streaming-rate UX).
    //
    //   - **Non-streaming-origin entry** (no captured fragments):
    //     re-route the cached text through fresh ReasoningSplitter +
    //     ToolCallSplitter, drain both at end-of-stream (Wave-3.5
    //     HIGH-2 fix at engine.rs:4332).  Preserves structural shape
    //     (Content / Reasoning / ToolCallDelta) but loses per-token
    //     boundaries.  This branch fires when the cache entry came
    //     from `generate_once_with_soft_tokens` (no per-token trace
    //     exists at non-streaming origin).
    if let Some((cached, cached_frags)) = loaded
        .prompt_cache
        .lookup_with_fragments(prompt_tokens, params)
    {
        tracing::debug!(
            "prompt_cache: STREAMING HIT — {} tokens served from cache, \
             prefill+decode skipped, fragments_branch={}",
            cached.prompt_tokens,
            cached_frags.is_some(),
        );
        if replay_cached_streaming_response_with_fragments(
            &cached,
            registration,
            params.tool_call_policy,
            events,
            cached_frags,
        )
        .is_err()
        {
            tracing::info!("SSE stream dropped by client during cache replay; aborting");
            if let Some(c) = cancellation_counter {
                c.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
            }
        }
        return;
    }

    // ── ADR-017 Phase E option (a) — streaming-path LCP probe ──
    //
    // Mirrors `generate_once_with_soft_tokens`'s probe + iter-3
    // resume gate. See that site for the full design contract.
    let resume_lcp: Option<usize> = {
        let lcp_key = build_lcp_key_for_request(loaded, params);
        let detected = crate::serve::kv_persist::lcp_registry::probe_lcp_opportunity(
            &mut loaded.lcp_registry,
            &lcp_key,
            prompt_tokens,
            !soft_tokens.is_empty(),
        );
        if let Some(sink) = loaded.kv_metrics_sink.as_ref() {
            sink.record_lcp_probe(detected);
        }
        match detected {
            None => None,
            Some(_k_obs) => {
                // Q3 auto-disable: mirrors the non-streaming gate. The shared
                // std::sync::Once inside warn_lcp_resume_without_dense ensures
                // exactly one log line per process across both probe sites.
                // "gemma-hybrid-lcp" (2026-08-03): resumable substrates
                // = dense (HF2Q_USE_DENSE=1) OR production hybrid. The
                // HB-encoded opt-out regime stays auto-disabled.
                let lcp_enabled = effective_kv_lcp_resume(
                    crate::debug::INVESTIGATION_ENV.kv_lcp_resume,
                    crate::debug::INVESTIGATION_ENV.use_dense
                        || crate::debug::INVESTIGATION_ENV.hybrid_kv,
                );
                if !lcp_enabled {
                    None
                } else {
                    let prefix_opt = loaded.lcp_registry.take_prefix(&lcp_key, prompt_tokens);
                    match prefix_opt {
                        None => None,
                        Some(prefix) => {
                            let new_linear = prompt_tokens.len() + params.max_tokens.max(1);
                            let model_sw = loaded.weights.sliding_window.max(1);
                            let agg_ok = prefix.linear_capacity >= new_linear
                                && prefix.sliding_window == model_sw;
                            // Per-layer cap + is_sliding check (same
                            // shape as non-streaming probe site).
                            let per_layer_ok = if !agg_ok {
                                false
                            } else if prefix.dense_kvs.len() != loaded.weights.layers.len() {
                                false
                            } else {
                                // ADR-017 Phase E.a iter-3.5a — dtype
                                // invariant added to the per-layer
                                // check. Model-current `kv_dtype` is
                                // resolved from `INVESTIGATION_ENV.f16_kv`
                                // (same source used at every alloc
                                // site). A cached entry with mismatched
                                // dtype must NOT be installed: the
                                // kernel's flash_attn_vec dispatch
                                // takes dtype as a static branch and
                                // would silently misread the cached
                                // bytes.
                                let model_kv_dtype = if crate::debug::INVESTIGATION_ENV.f16_kv {
                                    mlx_native::DType::F16
                                } else {
                                    mlx_native::DType::F32
                                };
                                // ADR-017 Phase E.a iter-3.6 follow-up
                                // (Codex audit LOW #1): align per-layer
                                // sliding required_cap with the alloc
                                // formula. When LONG_RESUME=1, sliding
                                // layers were allocated with
                                // `max(sw, new_linear)`; the per-layer
                                // check must demand ≥ same value, not
                                // just `model_sw`. Today the aggregate
                                // `prefix.linear_capacity >= new_linear`
                                // saves us, but a future refactor could
                                // admit an undersized sliding snapshot.
                                // "gemma-hybrid-lcp": long-resume admits
                                // dense OR production hybrid (kernel
                                // mask_type=2 verified for both legs).
                                let lr_long = crate::debug::INVESTIGATION_ENV.kv_lcp_long_resume
                                    && crate::debug::INVESTIGATION_ENV.kv_lcp_resume
                                    && (crate::debug::INVESTIGATION_ENV.use_dense
                                        || crate::debug::INVESTIGATION_ENV.hybrid_kv);
                                prefix.dense_kvs.iter().enumerate().all(|(li, arc)| {
                                    let layer = &loaded.weights.layers[li];
                                    let layer_is_ring = matches!(
                                        layer.layer_type,
                                        crate::serve::config::LayerType::Sliding
                                    );
                                    let required_cap = if layer_is_ring {
                                        if lr_long {
                                            model_sw.max(new_linear)
                                        } else {
                                            model_sw
                                        }
                                    } else {
                                        new_linear
                                    };
                                    // "gemma-hybrid-lcp" (2026-08-03):
                                    // the per-layer check runs on the
                                    // DENSE leg (prefill SDPA reads it);
                                    // the dense fields live behind
                                    // `arc.dense()` in the enum payload.
                                    let d = arc.dense();
                                    let dense_ok = d.capacity >= required_cap
                                        && d.is_sliding == layer_is_ring
                                        && d.dtype == model_kv_dtype;
                                    // Regime-consistency: under the
                                    // production hybrid regime the entry
                                    // MUST carry the hybrid leg per
                                    // layer — a dense-only entry under
                                    // hybrid would leave the decode cache
                                    // unrestored (silent zero-prefix; the
                                    // class this sub-iter exists to close).
                                    let regime_ok = if crate::debug::INVESTIGATION_ENV.hybrid_kv {
                                        match arc.hybrid() {
                                            Some(h) => {
                                                h.capacity >= required_cap
                                                    && h.is_sliding == layer_is_ring
                                            }
                                            None => false,
                                        }
                                    } else {
                                        true
                                    };
                                    dense_ok && regime_ok
                                })
                            };
                            if !per_layer_ok {
                                tracing::debug!(
                                    "lcp_resume (streaming): capacity check failed — falling back"
                                );
                                drop(prefix);
                                None
                            } else {
                                let k = prefix.k;
                                // "gemma-hybrid-lcp" (2026-08-03): split
                                // enum payload into dense-leg Arc install +
                                // hybrid-leg OWNED install (mirrors the
                                // non-streaming site; Arc exclusivity
                                // precondition + graceful bail on
                                // contention).
                                let mut dense_arcs: Vec<
                                    std::sync::Arc<
                                        crate::inference::models::gemma4::DenseKvBuffers,
                                    >,
                                > = Vec::with_capacity(prefix.dense_kvs.len());
                                let mut hybrid_owned: Vec<
                                    crate::inference::models::gemma4::HybridKvBuffers,
                                > = Vec::new();
                                let mut install_ok = true;
                                for arc in prefix.dense_kvs.into_iter() {
                                    match std::sync::Arc::try_unwrap(arc) {
                                        Ok(layer) => match layer {
                                            crate::inference::models::gemma4::GemmaLcpLayerKv::Dense(
                                                d,
                                            ) => {
                                                dense_arcs.push(std::sync::Arc::new(d));
                                            }
                                            crate::inference::models::gemma4::GemmaLcpLayerKv::DenseAndHybrid(
                                                d,
                                                h,
                                            ) => {
                                                dense_arcs.push(std::sync::Arc::new(d));
                                                hybrid_owned.push(h);
                                            }
                                        },
                                        Err(arc) => {
                                            tracing::debug!(
                                                "gemma-hybrid-lcp: payload Arc unexpectedly \
                                                 shared at install (strong_count={}) — fresh prefill",
                                                std::sync::Arc::strong_count(&arc)
                                            );
                                            install_ok = false;
                                            break;
                                        }
                                    }
                                }
                                if !install_ok {
                                    drop(dense_arcs);
                                    drop(hybrid_owned);
                                    None
                                } else {
                                    let has_hybrid = !hybrid_owned.is_empty();
                                    loaded.weights.dense_kvs = Some(dense_arcs);
                                    if has_hybrid {
                                        loaded.weights.hybrid_kv = Some(hybrid_owned);
                                    }
                                    tracing::debug!(
                                        "lcp_resume (streaming): ENGAGED — K={} of N={}",
                                        k,
                                        prompt_tokens.len(),
                                    );
                                    Some(k)
                                }
                            }
                        }
                    }
                }
            }
        }
    };

    // Reasoning splitter — classifies each decoded fragment into the
    // content / reasoning_content slot. `None` when the model has no
    // registered reasoning markers; all fragments then route to `Content`.
    let mut splitter = registration
        .and_then(|r| super::registry::make_reasoning_splitter(r, params.reasoning_forced_open));

    // Tool-call splitter (iter-133 Iter B-2) — classifies the
    // post-reasoning Content stream into in/out-of-tool-call spans. When a
    // tool-call span closes, its body is parsed into structured
    // `name + arguments_json` and emitted as one or more
    // `GenerationEvent::ToolCallDelta` chunks (id+name first, full
    // arguments string second; matches the SSE encoder's expectation in
    // `sse.rs:208-247`).
    //
    // Composition: the engine runs ReasoningSplitter first; any
    // `Content`-classified output then flows into ToolCallSplitter. Reasoning
    // never appears inside a tool call — neither chat template emits a
    // reasoning-open marker (Gemma 4 `<|channel>` or Qwen 3.5/3.6 `<think>`)
    // between tool-call markers — so this layering is safe.
    let mut tool_splitter =
        registration.and_then(|r| super::registry::ToolCallSplitter::from_registration(r));
    // Per-call body accumulator + per-stream tool-call index. Body is
    // bounded by max_tokens so unbounded growth is impossible. Index is
    // incremented every time a tool-call closes and emits a delta — used
    // as the OpenAI `delta.tool_calls[*].index` field.
    let mut tool_call_body: String = String::new();
    let mut tool_call_index: usize = 0;
    // Set true on first ToolCallClose; latched. Drives `finish_reason ==
    // "tool_calls"` per OpenAI spec (decode loop's normal `"stop"` /
    // `"length"` is overridden when this flag is set on the terminating
    // path).
    let mut saw_tool_call: bool = false;

    // Wave 3 W-B3 — T2.3 incremental tool-call argument streaming.
    //
    // Per-call streaming-emit state. `Some(...)` between `ToolCallOpen` and
    // `ToolCallClose`; `None` otherwise. `ToolCallText` fragments call
    // `emitter.advance(body, events)` to flush newly-extractable name + kv
    // fragments immediately rather than waiting for `ToolCallClose` to
    // emit one big arguments delta. `ToolCallClose` calls
    // `emitter.finalize(...)` which either:
    //   - emits the closing `}` + any tail kvs (streaming path was active), or
    //   - delegates to `emit_streaming_tool_call_close` (streaming path
    //     declined, falling back to pre-W-B3 close-buffered shape).
    //
    // Cache replay keeps the close-buffered path — see `route_replay_fragment`.
    // Incremental emission has zero benefit for cache hits (the full text is
    // available synchronously) and would force two divergent SSE shapes for
    // identical `cached.text` content.
    let mut tool_call_emitter: Option<ToolCallStreamEmitter> = None;

    // Wave-2.5 A4: capture the policy so the route_content closure can branch
    // on Constrained vs Auto when a tool-call body fails to parse.
    let tool_call_policy = params.tool_call_policy;

    // Wave 2.6 W-α5 Q2: the wave-2.5 `Arc<AtomicBool> grammar_active`
    // sibling-state pattern is REMOVED.  The trigger gate now lives
    // inside `GrammarRuntime` itself (`is_awaiting_trigger()`); the
    // `route_content` closure flips it via `runtime.trigger()` on
    // ToolCallOpen, and the decode loop calls `mask_invalid_tokens` /
    // `accept_bytes` / `is_dead` UNCONDITIONALLY — all three self-gate
    // on the SAME boolean, eliminating the split-state condition the
    // wave-2.5 audit caught at engine.rs:1401, 1489, 1554, 2041, 2145,
    // 2195.  See cfa-20260427-adr005-wave2.6 research-report.md Q2 +
    // /opt/llama.cpp/src/llama-grammar.cpp:1287-1439 for the canonical
    // pattern.
    //
    // To call `runtime.trigger()` from the `route_content` closure, the
    // closure needs mutable access to the runtime.  The runtime is
    // owned by `generate_stream_once`, so we share it via
    // `Rc<RefCell<...>>` — single-threaded, no atomics needed (the
    // streaming worker is one OS thread).  When grammar is None,
    // `grammar_runtime` is None and the trigger plumbing is a no-op.
    // (We use `Rc<RefCell<...>>` instead of an `&mut` borrow because
    // the closure outlives the borrow checker's view of the runtime
    // through the decode loop — same shape used elsewhere in this
    // function for shared mutable state.)

    // Helper: for a Content-classified text run, route through the
    // ToolCallSplitter (if any) and emit the appropriate
    // GenerationEvent. When ToolCallSplitter is None, route every byte to
    // `DeltaKind::Content` (current behavior pre-iter-B-2).
    //
    // Wave 2.6 W-α5 Q2: takes `grammar_runtime: &mut Option<GrammarRuntime>`
    // so the ToolCallOpen branch can call `runtime.trigger()` — this is
    // the splice point where the lazy-grammar awakens.  ToolCallClose
    // does NOT reset (multi-call grammars rely on the grammar shape
    // accepting `(call)+`; see research-report.md Q2 + llama.cpp PR
    // #9639).
    let route_content = |tool_splitter: &mut Option<super::registry::ToolCallSplitter>,
                         body: &mut String,
                         tc_index: &mut usize,
                         saw_tc: &mut bool,
                         emitter: &mut Option<ToolCallStreamEmitter>,
                         grammar_runtime: &mut Option<super::grammar::GrammarRuntime>,
                         events: &EventSink<'_>,
                         text: &str,
                         reg: Option<&super::registry::ModelRegistration>|
     -> Result<(), ()> {
        if text.is_empty() {
            return Ok(());
        }
        let Some(tcs) = tool_splitter.as_mut() else {
            // No tool markers registered — original behavior.
            if events
                .blocking_send(GenerationEvent::Delta {
                    kind: DeltaKind::Content,
                    text: text.to_string(),
                })
                .is_err()
            {
                return Err(());
            }
            return Ok(());
        };
        for ev in tcs.feed(text) {
            match ev {
                super::registry::ToolCallEvent::Content(t) => {
                    if !t.is_empty()
                        && events
                            .blocking_send(GenerationEvent::Delta {
                                kind: DeltaKind::Content,
                                text: t,
                            })
                            .is_err()
                    {
                        return Err(());
                    }
                }
                super::registry::ToolCallEvent::ToolCallOpen => {
                    body.clear();
                    // Wave 3 W-B3 — T2.3 incremental: construct a fresh
                    // emitter for this call. Family from registration; an
                    // unregistered family yields an emitter that declines
                    // (its `advance` is a no-op and `finalize` falls back
                    // to `emit_streaming_tool_call_close`).
                    *emitter = Some(ToolCallStreamEmitter::new(reg.map(|r| r.family), *tc_index));
                    // Wave 2.6 W-α5 Q2: entering a tool-call body — flip
                    // the grammar runtime's trigger so subsequent decode
                    // steps enforce the body grammar.  No-op when the
                    // runtime is None (no grammar request) or already
                    // post-trigger (re-entry on a grammar without
                    // explicit reset support — llama.cpp behavior).
                    if let Some(rt) = grammar_runtime.as_mut() {
                        rt.trigger();
                    }
                }
                super::registry::ToolCallEvent::ToolCallText(t) => {
                    body.push_str(&t);
                    // Wave 3 W-B3 — T2.3 incremental: drive the per-call
                    // emitter to flush newly-extractable name + kv
                    // fragments. No-op when the family is unregistered
                    // (`advance` returns Ok(()) without sending anything,
                    // leaving finalize to use the close-buffered fallback).
                    if let Some(em) = emitter.as_mut() {
                        em.advance(body, events)?;
                    }
                }
                super::registry::ToolCallEvent::ToolCallClose => {
                    // Wave 2.6 W-α5 Q2: leaving a tool-call body.  The
                    // grammar runtime is NOT reset — single-call termination
                    // is delivered by the grammar shape exhausting (iter-218:
                    // `parallel_tool_calls=false` default → shape `body close
                    // space` exhausts after first close → is_dead → halt).
                    // Multi-call mode (`parallel_tool_calls=true` opt-in)
                    // relies on the `gemma4-call*` recursion accepting
                    // subsequent open markers (Hermes 2 Pro template; see
                    // `/opt/llama.cpp/common/chat.cpp:1399-1416` `p.repeat`).
                    //
                    // Wave 3 W-A3: close-time dispatch delegated to
                    // `emit_streaming_tool_call_close` so the parse-failure
                    // branch can be audit-driver tested independently.
                    //
                    // Wave 3 W-B3: if the per-call emitter activated mid-
                    // stream (name was emitted), finalize emits the
                    // closing `}` + tail kvs and increments `tc_index`.
                    // Otherwise finalize delegates to the legacy
                    // `emit_streaming_tool_call_close` so the close-
                    // buffered shape AND policy-enforced loud-error
                    // branches stay byte-for-byte identical.
                    let body_dump = std::mem::take(body);
                    let mut em = emitter.take().unwrap_or_else(|| {
                        // Defensive — ToolCallOpen always precedes Close,
                        // but if a buggy splitter ever emits Close-without-
                        // Open we still want the legacy close-buffered
                        // semantics to fire.
                        ToolCallStreamEmitter::new(reg.map(|r| r.family), *tc_index)
                    });
                    em.finalize(body_dump, reg, tool_call_policy, tc_index, saw_tc, events)?;
                }
            }
        }
        Ok(())
    };

    // Local helper to emit a fragment through the reasoning splitter (if
    // any) and then through the tool-call router. Returns the bytes emitted
    // (for stop-string bookkeeping). Note: each splitter holds back a tail
    // that's drained at generation end.
    let emit_fragment = |splitter: &mut Option<super::registry::ReasoningSplitter>,
                         tool_splitter: &mut Option<super::registry::ToolCallSplitter>,
                         body: &mut String,
                         tc_index: &mut usize,
                         saw_tc: &mut bool,
                         emitter: &mut Option<ToolCallStreamEmitter>,
                         grammar_runtime: &mut Option<super::grammar::GrammarRuntime>,
                         events: &EventSink<'_>,
                         fragment: &str,
                         reg: Option<&super::registry::ModelRegistration>|
     -> Result<(), ()> {
        if fragment.is_empty() {
            return Ok(());
        }
        if let Some(sp) = splitter.as_mut() {
            for (slot, text) in sp.feed(fragment) {
                match slot {
                    super::registry::SplitSlot::Reasoning => {
                        if !text.is_empty()
                            && events
                                .blocking_send(GenerationEvent::Delta {
                                    kind: DeltaKind::Reasoning,
                                    text,
                                })
                                .is_err()
                        {
                            return Err(());
                        }
                    }
                    super::registry::SplitSlot::Content => {
                        route_content(
                            tool_splitter,
                            body,
                            tc_index,
                            saw_tc,
                            emitter,
                            grammar_runtime,
                            events,
                            &text,
                            reg,
                        )?;
                    }
                }
            }
        } else {
            // No reasoning splitter — route everything as Content.
            route_content(
                tool_splitter,
                body,
                tc_index,
                saw_tc,
                emitter,
                grammar_runtime,
                events,
                fragment,
                reg,
            )?;
        }
        Ok(())
    };

    // Snapshot mlx-native process-global GPU counters pre-generation so we
    // can report the per-request delta on the terminal `Done` event's
    // StreamStats (mirrors the non-streaming path's x_hf2q_timing counters).
    let pre_dispatches = mlx_native::dispatch_count();
    let pre_syncs = mlx_native::sync_count();

    // ── Sampler config — Tier 2/3/4 + grammar (iter-94 / iter-95, mirrors generate_once) ──
    let sample_logits = params.temperature > 0.0
        || params.top_k > 0
        || params.top_p < 1.0
        || params.repetition_penalty != 1.0
        || !params.logit_bias.is_empty()
        || params.grammar.is_some();
    let sampler_params = if sample_logits {
        Some(SamplerParams {
            temperature: params.temperature as f64,
            top_p: params.top_p as f64,
            top_k: params.top_k,
            min_p: 0.0,
            repetition_penalty: effective_repetition_penalty(params),
            max_tokens: params.max_tokens,
        })
    } else {
        None
    };
    // Wave 2.6 W-α5 Q2 + Wave 2.7 W-η Q-A: arm the trigger gate ONLY when
    // the request carries an AUTO-mode tool-call body grammar
    // (`GrammarKind::ToolCallBodyAuto`).  `ResponseFormat` and
    // `ToolCallBodyRequired` runtimes leave the gate disarmed for eager
    // enforcement from token 0 — fixes audit divergence A1 /
    // response_format regression for ResponseFormat, and forces tool-call
    // emission for Required/Function (mirrors llama.cpp grammar_lazy=false
    // in common/chat.cpp:898-913, 1177-1200, 1399-1416).
    let mut grammar_runtime: Option<super::grammar::GrammarRuntime> = match params.grammar.as_ref()
    {
        Some(g) => {
            let start_rule_id = match g.rule_id("root") {
                Some(id) => id,
                None => {
                    send!(GenerationEvent::Error("grammar has no root rule".into()));
                    return;
                }
            };
            match super::grammar::GrammarRuntime::new(g.clone(), start_rule_id) {
                Some(mut rt) => {
                    // Wave 2.7 W-η Q-A: see non-streaming arming above —
                    // `ToolCallBodyRequired` keeps the eager gate
                    // (`awaiting_trigger=false`); only `ToolCallBodyAuto`
                    // suspends until ToolCallSplitter sees the open marker.
                    if matches!(params.grammar_kind, GrammarKind::ToolCallBodyAuto) {
                        rt.set_awaiting_trigger(true);
                    }
                    Some(rt)
                }
                None => {
                    send!(GenerationEvent::Error("grammar runtime init failed".into()));
                    return;
                }
            }
        }
        None => None,
    };
    let token_bytes_ref: Option<&[Vec<u8>]> = params.token_bytes.as_deref().map(|v| &v[..]);

    // --- Prefill ---
    // Iter-211 W79: routed through `forward_prefill_with_soft_tokens` so
    // vision content parts can stream. `soft_tokens` is empty for text-only
    // requests; the prefill API treats an empty slice as identity over
    // `forward_prefill` (`src/serve/forward_prefill.rs:117`), so the
    // text-only path stays byte-identical.
    // ADR-028 iter-415: same batched-prefill opt-in as non-streaming
    // path above.  Streaming wraps in a Result-match; batched path
    // bails on its own anyhow::Result so we propagate via the same
    // match arm.
    let prefill_start = Instant::now();
    // ADR-028 iter-421 default-flipped: per iter-326 operator REFRAME #2
    // ("default should have the best things on that provide the best
    // mantra-aligned outcome for users").  Phase 15 has been validated 4x:
    // iter-415 short prompts byte-identical, iter-416 multi-turn coherent,
    // iter-420 pp3.4K byte-identical, iter-421 long-decode/sampling/
    // streaming all robust.  Opt out via `HF2Q_SERVE_BATCHED_PREFILL=0`
    // / `=false` / `=off` (matches iter-326 q6_K_NR2 default-on pattern).
    // Tri-state (2026-08-03 auto-fallback): explicit =1 FORCES the
    // batched route (operator override); explicit =0/=false/=off forces
    // the linear route; UNSET = auto — engage batched only when this
    // request's O(n²) mask overhead fits the available-memory budget.
    // A 92K-token opencode first turn allocated ~120 GB transient on
    // 2026-08-03 and died in Metal with a command-buffer error — no
    // user should need to know BATCHED=0 exists.
    let serve_batched_env = std::env::var("HF2Q_SERVE_BATCHED_PREFILL").ok();
    let batched_allowed = match serve_batched_env.as_deref() {
        Some(v) => !matches!(v.to_ascii_lowercase().as_str(), "0" | "false" | "off"),
        None => {
            let viable = crate::serve::forward_prefill_batched::serve_batched_route_viable(
                prompt_tokens.len(),
                loaded.weights.num_attention_heads,
            );
            if !viable {
                eprintln!(
                    "[hf2q batched prefill] auto-fallback to linear route: \
                     seq_len={} O(n²) mask overhead exceeds the available- \
                     memory budget (force-on with HF2Q_SERVE_BATCHED_PREFILL=1)",
                    prompt_tokens.len()
                );
            }
            viable
        }
    };
    let use_batched_serve = soft_tokens.is_empty() && resume_lcp.is_none() && batched_allowed;
    let next_token_result = if use_batched_serve {
        loaded
            .weights
            .forward_prefill_batched(prompt_tokens, max_tokens, 0, &mut loaded.ctx)
    } else {
        loaded.weights.forward_prefill_with_soft_tokens_resume(
            prompt_tokens,
            soft_tokens,
            max_tokens,
            &mut loaded.ctx,
            resume_lcp,
            false, // slot_aware=false (ADR-040 STEP-1b): legacy byte-equivalent
        )
    };
    let prefill_duration = prefill_start.elapsed();
    let prefill_argmax = match next_token_result {
        Ok(t) => t,
        Err(e) => {
            send!(GenerationEvent::Error(format!("prefill failed: {e}")));
            return;
        }
    };
    // First decode token: greedy fast-path uses prefill's on-GPU argmax;
    // sampling path re-derives from prefill's live logits (last
    // prompt-token's lm_head output) so user-controlled temperature
    // applies to the very first generated token.
    let mut next_token = if let Some(sp) = sampler_params.as_ref() {
        let logits_view = match loaded.weights.logits_view() {
            Ok(v) => v.to_vec(),
            Err(e) => {
                send!(GenerationEvent::Error(format!(
                    "first-token logits read: {e}"
                )));
                return;
            }
        };
        let mut logits = logits_view;
        if !params.logit_bias.is_empty() {
            let v = logits.len();
            for (&id, &bias) in &params.logit_bias {
                let idx = id as usize;
                if idx < v {
                    logits[idx] += bias;
                }
            }
        }
        // Wave 2.6 W-α5 Q2: mask + accept are UNCONDITIONAL.  For
        // `ToolCallBody`-kind runtimes the gate is armed (no-op);
        // for `ResponseFormat`-kind it enforces from token 0 — the
        // wave-2.5 audit fix for the response_format regression.
        if let (Some(rt), Some(tb)) = (grammar_runtime.as_ref(), token_bytes_ref) {
            super::grammar::mask::mask_invalid_tokens(rt, tb, &mut logits);
        }
        let tok = sampler_pure::sample_token(&mut logits, sp, &[]);
        if let (Some(rt), Some(tb)) = (grammar_runtime.as_mut(), token_bytes_ref) {
            let bytes = tb.get(tok as usize).map(|v| v.as_slice()).unwrap_or(&[]);
            if !bytes.is_empty() {
                rt.accept_bytes(bytes);
            }
        }
        tok
    } else {
        prefill_argmax
    };

    // --- Decode loop ---
    let decode_start = Instant::now();
    let mut completion_tokens = 0usize;
    let mut accumulated_text = String::new();
    let mut reasoning_token_count = 0usize;
    let mut finish_reason: &'static str = "length";
    let mut profiler = ProfileAccumulator::new(0);
    // Iter-94: streaming path needs the running token list for
    // sampler_pure's repetition_penalty.  Pre-iter-94 only the
    // accumulated_text was tracked (sufficient for stop-string scan),
    // because the loop ran greedy-only.
    let mut generated_tokens: Vec<u32> = Vec::with_capacity(max_tokens);
    generated_tokens.push(next_token);

    // ADR-017 Phase E.a iter-3 + Codex Phase-2b audit (streaming
    // mirror): physical decode-write counter — see the matching
    // declaration in `generate_once_with_soft_tokens` for full
    // rationale. Used by the post-decode LCP store to decide whether
    // the sliding ring wrapped (which would corrupt cached prompt-
    // prefix state).
    let mut physical_decode_writes: usize = 0;

    // Emit prefill-produced first token:
    let first_text = loaded
        .tokenizer
        .decode(&[next_token], false)
        .unwrap_or_default();
    let mut is_eos_first = loaded.eos_token_ids.contains(&next_token);
    if !is_eos_first && !first_text.is_empty() {
        accumulated_text.push_str(&first_text);
        if emit_fragment(
            &mut splitter,
            &mut tool_splitter,
            &mut tool_call_body,
            &mut tool_call_index,
            &mut saw_tool_call,
            &mut tool_call_emitter,
            &mut grammar_runtime,
            events,
            &first_text,
            registration,
        )
        .is_err()
        {
            tracing::info!("SSE stream dropped by client; aborting decode");
            return;
        }
    }
    completion_tokens += 1;
    if splitter.as_ref().map(|s| s.in_reasoning()).unwrap_or(false) {
        reasoning_token_count += 1;
    }
    if is_eos_first {
        finish_reason = "stop";
    } else if hit_stop_string(&accumulated_text, &params.stop_strings) {
        finish_reason = "stop";
        is_eos_first = true;
    }

    if !is_eos_first {
        for _ in 1..max_tokens {
            let pos = prompt_len + completion_tokens - 1;
            let mut p = profiler.start_token();
            let dec_result =
                loaded
                    .weights
                    .forward_decode(next_token, pos, &mut loaded.ctx, &mut p);
            profiler.finish_token(p);
            let greedy_token = match dec_result {
                Ok(t) => t,
                Err(e) => {
                    send!(GenerationEvent::Error(format!("decode failed: {e}")));
                    return;
                }
            };
            // ADR-017 Phase E.a iter-3 — count physical KV write
            // (mirrors non-streaming counter). One per `forward_decode`
            // success; counted BEFORE any subsequent EOS / stop_string
            // / grammar-dead branches.
            physical_decode_writes += 1;
            next_token = if let Some(sp) = sampler_params.as_ref() {
                let mut logits: Vec<f32> = match loaded.weights.logits_view() {
                    Ok(v) => v.to_vec(),
                    Err(e) => {
                        send!(GenerationEvent::Error(format!("logits read: {e}")));
                        return;
                    }
                };
                if !params.logit_bias.is_empty() {
                    let v = logits.len();
                    for (&id, &bias) in &params.logit_bias {
                        let idx = id as usize;
                        if idx < v {
                            logits[idx] += bias;
                        }
                    }
                }
                // Wave 2.6 W-α5 Q2: mask + accept UNCONDITIONAL.  The
                // runtime self-gates on `is_awaiting_trigger()`.  When
                // suspended (ToolCallBody pre-trigger), mask is a no-op
                // and the model emits preamble freely; once
                // `route_content` sees the open marker and calls
                // `runtime.trigger()`, every subsequent step enforces.
                // ResponseFormat-kind runtimes were never suspended and
                // enforce from the first token.  This collapses the
                // wave-2.5 4-line `if grammar_active.load { mask }` /
                // separate `accept` paired pattern into 2 lines that are
                // structurally correct.
                if let (Some(rt), Some(tb)) = (grammar_runtime.as_ref(), token_bytes_ref) {
                    super::grammar::mask::mask_invalid_tokens(rt, tb, &mut logits);
                }
                let tok = sampler_pure::sample_token(&mut logits, sp, &generated_tokens);
                if let (Some(rt), Some(tb)) = (grammar_runtime.as_mut(), token_bytes_ref) {
                    let bytes = tb.get(tok as usize).map(|v| v.as_slice()).unwrap_or(&[]);
                    if !bytes.is_empty() {
                        rt.accept_bytes(bytes);
                    }
                }
                tok
            } else {
                greedy_token
            };

            if loaded.eos_token_ids.contains(&next_token) {
                finish_reason = "stop";
                break;
            }
            completion_tokens += 1;
            generated_tokens.push(next_token);
            let fragment = loaded
                .tokenizer
                .decode(&[next_token], false)
                .unwrap_or_default();
            accumulated_text.push_str(&fragment);
            if emit_fragment(
                &mut splitter,
                &mut tool_splitter,
                &mut tool_call_body,
                &mut tool_call_index,
                &mut saw_tool_call,
                &mut tool_call_emitter,
                &mut grammar_runtime,
                events,
                &fragment,
                registration,
            )
            .is_err()
            {
                tracing::info!("SSE stream dropped by client; aborting decode");
                return;
            }
            if splitter.as_ref().map(|s| s.in_reasoning()).unwrap_or(false) {
                reasoning_token_count += 1;
            }
            if hit_stop_string(&accumulated_text, &params.stop_strings) {
                finish_reason = "stop";
                break;
            }
            // Grammar-driven termination — see generate_once for full doc.
            // Streaming variant: we can't pop the trailing token cleanly
            // because the fragment was already emitted to the SSE stream;
            // accept the small wart.  Iter-96+ candidate: hold back the
            // last fragment until next-step grammar state is known so it
            // can be suppressed pre-emit.
            if grammar_runtime.as_ref().is_some_and(|rt| rt.is_dead()) {
                finish_reason = "stop";
                break;
            }
            if let Some(rt) = grammar_runtime.as_ref() {
                if rt.is_accepted() {
                    if let Some(tb) = token_bytes_ref {
                        let bytes = tb
                            .get(next_token as usize)
                            .map(|v| v.as_slice())
                            .unwrap_or(&[]);
                        if bytes.is_empty() {
                            finish_reason = "stop";
                            break;
                        }
                    }
                }
            }
        }
    }

    // Drain any leftover tail the reasoning splitter was holding back. If
    // the tail is Content-classified, route it through the tool-call
    // splitter so a marker straddling EOS is still detected.
    if let Some(sp) = splitter.as_mut() {
        if let Some((slot, tail)) = sp.finish() {
            match slot {
                super::registry::SplitSlot::Reasoning => {
                    if !tail.is_empty()
                        && events
                            .blocking_send(GenerationEvent::Delta {
                                kind: DeltaKind::Reasoning,
                                text: tail,
                            })
                            .is_err()
                    {
                        tracing::info!("SSE stream dropped by client; aborting decode");
                        return;
                    }
                }
                super::registry::SplitSlot::Content => {
                    if route_content(
                        &mut tool_splitter,
                        &mut tool_call_body,
                        &mut tool_call_index,
                        &mut saw_tool_call,
                        &mut tool_call_emitter,
                        &mut grammar_runtime,
                        events,
                        &tail,
                        registration,
                    )
                    .is_err()
                    {
                        tracing::info!("SSE stream dropped by client; aborting decode");
                        return;
                    }
                }
            }
        }
    }
    // Drain any tool-splitter tail and then enforce Constrained-policy
    // safety nets before Done. The drain + check logic is factored into
    // `finalize_streaming_tool_state` so the test harness can drive the
    // exact production code path (audit-driver test for HIGH-1; do not
    // duplicate this logic in a test stand-in).
    match finalize_streaming_tool_state(
        tool_splitter.as_mut(),
        tool_call_policy,
        saw_tool_call,
        registration,
        completion_tokens,
        accumulated_text.len(),
        events,
    ) {
        FinalizeStreamingAction::Continue => {}
        FinalizeStreamingAction::ClientDropped => {
            tracing::info!("SSE stream dropped by client; aborting decode");
            return;
        }
        FinalizeStreamingAction::ErrorEmitted => {
            // Mid-call truncation or no-call under Constrained already
            // emitted GenerationEvent::Error — do NOT also emit Done. The
            // SSE encoder closes the stream with a finish_reason="error"
            // final chunk on receipt of Error (sse.rs:298-322).
            return;
        }
    }

    // Override finish_reason to "tool_calls" per OpenAI spec when at least
    // one structured tool-call delta was emitted on this stream. Spec:
    // https://platform.openai.com/docs/guides/function-calling — when the
    // model invokes a tool, finish_reason becomes "tool_calls" rather than
    // "stop"/"length". This overrides EOS-driven "stop" set above.
    if saw_tool_call {
        finish_reason = "tool_calls";
    }

    let decode_duration = decode_start.elapsed();

    let stats = StreamStats {
        prefill_time_secs: Some(prefill_duration.as_secs_f64()),
        decode_time_secs: Some(decode_duration.as_secs_f64()),
        total_time_secs: Some((prefill_duration + decode_duration).as_secs_f64()),
        time_to_first_token_ms: Some(prefill_duration.as_secs_f64() * 1000.0),
        prefill_tokens_per_sec: Some(if prefill_duration.as_secs_f64() > 0.0 {
            prompt_len as f64 / prefill_duration.as_secs_f64()
        } else {
            0.0
        }),
        decode_tokens_per_sec: Some(if decode_duration.as_secs_f64() > 0.0 {
            completion_tokens as f64 / decode_duration.as_secs_f64()
        } else {
            0.0
        }),
        gpu_sync_count: Some(mlx_native::sync_count().saturating_sub(pre_syncs)),
        gpu_dispatch_count: Some(mlx_native::dispatch_count().saturating_sub(pre_dispatches)),
        cached_prompt_tokens: None,
        reasoning_tokens: if reasoning_token_count > 0 {
            Some(reasoning_token_count)
        } else {
            None
        },
    };

    send!(GenerationEvent::Done {
        finish_reason,
        prompt_tokens: prompt_len,
        completion_tokens,
        stats,
    });

    // Iter-96 prompt cache update on streaming completion.
    //
    // Wave 3 W-A2: the streaming path now ALSO consults the cache on
    // input via `replay_cached_streaming_response` (see lookup at the
    // top of this function).  The store here is unchanged; updating on
    // every successful completion lets BOTH a subsequent streaming AND a
    // subsequent non-streaming request with the same prompt+params hit
    // the cache (the cache slot is mode-agnostic; only the lookup +
    // replay path differs).
    //
    // Cache-shape note: `text` is set to the full pre-split
    // `accumulated_text` (markers and all).  The replay helper re-routes
    // through fresh ReasoningSplitter + ToolCallSplitter to re-emit the
    // proper SSE shape, so embedded markers re-classify correctly.
    // `reasoning_text: None` because the live splitter already routed
    // reasoning fragments into Reasoning deltas during decode — there is
    // no separately-tracked reasoning string to replay (the assembled
    // text alone, fed back through a fresh splitter, reproduces the
    // same event sequence on hit).
    let cache_result = GenerationResult {
        text: accumulated_text.clone(),
        reasoning_text: None, // splitter already routed reasoning into Delta events
        prompt_tokens: prompt_len,
        completion_tokens,
        reasoning_tokens: if reasoning_token_count > 0 {
            Some(reasoning_token_count)
        } else {
            None
        },
        finish_reason,
        prefill_duration,
        decode_duration,
        cached_tokens: 0,
        logprobs: None,
    };

    // ADR-005 iter-224 W-A2.2: streaming-origin fragment capture.
    //
    // `captured_fragments` was populated by `EventSink::with_capture`
    // mirroring every `GenerationEvent::Delta` / `::ToolCallDelta`
    // forwarded to the SSE channel during the decode loop above (see
    // `let sink = EventSink::with_capture(...)` at function entry).
    // Pass it to `store_with_fragments` so a future cache hit on the
    // same prompt+params replays via the W-A2.3 fragments branch
    // (byte-identical event-stream framing).  `Some(...)` here is what
    // distinguishes streaming-origin entries from the non-streaming
    // origin path (`generate_once_with_soft_tokens` calls plain
    // `store(...)` which sets fragments=None — see Worker AA design
    // §3b option (a)).
    //
    // Drop the `sink` (and the `events` alias which borrows `sink`)
    // before `store_with_fragments`, which mutates
    // `loaded.prompt_cache` (siblings of the channel borrow); `sink`
    // borrows the channel sender only, so dropping it does not affect
    // `loaded`.  The captured vec is moved into the cache via
    // `RefCell::into_inner`.
    let _ = events; // release the `&sink` alias (was a reference; clippy: `drop` of reference)
    drop(sink);
    let fragments = captured_fragments.into_inner();
    loaded
        .prompt_cache
        .store_with_fragments(prompt_tokens, params, &cache_result, Some(fragments));

    // ADR-017 Phase E.a iter-3.5b (streaming origin): mirror the
    // non-streaming snapshot-store site. Take the end-of-prefill
    // snapshot (populated by `forward_prefill_with_soft_tokens_resume`
    // BEFORE decode mutated the live buffers) and register it.
    // Decode operated on the LIVE `weights.dense_kvs`, NOT the
    // snapshot, so the snapshot faithfully represents [0..N) of the
    // prompt. The wrap guard is no longer needed; long-conversation
    // prompts are cacheable.
    //
    // Skip multimodal per §10.5; skip when snapshot is None
    // (env-gates off / embedding-only path).
    let _ = physical_decode_writes; // retained debug counter
    if soft_tokens.is_empty() {
        // "gemma-hybrid-lcp" (2026-08-03): take the hybrid leg snapshot
        // alongside the dense one; both are populated at end-of-prefill
        // under the production hybrid regime (None otherwise).
        let hybrid_snapshot = loaded.weights.hybrid_kv_snapshot_for_lcp.take();
        if let Some(snapshot) = loaded.weights.dense_kvs_snapshot_for_lcp.take() {
            // "gemma-hybrid-lcp": build the regime-aware payload. On
            // fail-safe (layer mismatch / shared Arc) this is None and
            // the store below is skipped (clean future miss, never fatal).
            let payload = build_gemma_lcp_payload(snapshot, hybrid_snapshot);
            let sliding_window = loaded.weights.sliding_window.max(1);
            let has_sliding_layer = loaded
                .weights
                .layers
                .iter()
                .any(|l| matches!(l.layer_type, crate::serve::config::LayerType::Sliding));
            // iter-3.5c prefill-wrap guard (mirrors non-streaming).
            // ADR-017 Phase E.a iter-3.6: lift when LONG_RESUME=1 (mirrors
            // engine.rs:4516 non-streaming site).
            // "gemma-hybrid-lcp": long-resume admits dense OR production
            // hybrid (mirrors the probe-side gate).
            let kv_lcp_long_resume = crate::debug::INVESTIGATION_ENV.kv_lcp_long_resume
                && crate::debug::INVESTIGATION_ENV.kv_lcp_resume
                && (crate::debug::INVESTIGATION_ENV.use_dense
                    || crate::debug::INVESTIGATION_ENV.hybrid_kv);
            let prefill_safe =
                !has_sliding_layer || prompt_tokens.len() <= sliding_window || kv_lcp_long_resume;
            if prefill_safe {
                let lcp_key = build_lcp_key_for_request(loaded, params);
                // iter-3.5d headroom (mirrors non-streaming site).
                let linear_capacity =
                    sliding_window.max(prompt_tokens.len() + params.max_tokens.max(1));
                // Codex Phase-2b 2026-05-06: surface store errors instead
                // of `let _ = ...` so EntryExceedsBudget / EmptyPrompt /
                // EmptyPayload aren't swallowed silently. Mantra: no fallback.
                // "gemma-hybrid-lcp": store gated on the payload build
                // (None = fail-safe skip).
                if let Some(payload) = payload {
                    if let Err(e) = loaded.lcp_registry.store(
                        lcp_key,
                        prompt_tokens.to_vec(),
                        payload,
                        sliding_window,
                        linear_capacity,
                    ) {
                        tracing::warn!(
                            error = ?e,
                            "gemma streaming lcp_registry store failed"
                        );
                    }
                }
            } else {
                tracing::debug!(
                    "lcp_registry.store skipped (streaming): prefill-wrap \
                     guard (prompt_len={} > sliding_window={})",
                    prompt_tokens.len(),
                    sliding_window
                );
            }
        }
    }
}

fn hit_stop_string(text: &str, stops: &[String]) -> bool {
    if stops.is_empty() {
        return false;
    }
    stops
        .iter()
        .any(|s| !s.is_empty() && text.ends_with(s.as_str()))
}

fn strip_trailing_stop(text: &mut String, stops: &[String]) {
    for s in stops {
        if !s.is_empty() && text.ends_with(s) {
            let new_len = text.len() - s.len();
            text.truncate(new_len);
            return;
        }
    }
}

// ---------------------------------------------------------------------------
// Tokenizer + chat-template helpers usable from handlers
// ---------------------------------------------------------------------------

/// Render a Jinja2 chat template over an OpenAI-shaped message list.
///
/// The minijinja environment mirrors the one the one-shot `cmd_generate`
/// path uses: `messages`, `add_generation_prompt`, `bos_token`, `eos_token`,
/// and (when supplied) `tools` are in scope. Content handling:
///
///   - `content: "plain string"` → the template sees `content = "..."`.
///   - `content: [{type:"text", text:"..."}, ...]` → text parts are
///     concatenated; image parts are ignored in this iter (multimodal lands
///     with Phase 2c). A future iter will pass typed parts to vision-aware
///     templates.
///   - OpenAI `assistant` role is remapped to `model` if the GGUF template
///     is Gemma 4 (detected by presence of `<|turn>model` in the template).
///     Otherwise roles are passed through verbatim.
///   - Per-message `tool_calls` (assistant-emitted) and synthetic
///     `tool_responses` (synthesized from OpenAI `role: "tool"` history
///     messages, see [`render_chat_prompt_with_tools`]) are exposed to the
///     template as message fields so tool-aware templates (e.g. Gemma 4's
///     `<|tool_call>` / `<|tool_response>` markers) render correctly.
///
/// Use [`render_chat_prompt_with_tools`] to supply tool definitions; the
/// thin entry-point `render_chat_prompt` is kept for legacy callers (one-shot
/// `cmd_generate`, the chat-template overflow path) that don't carry tools.
pub fn render_chat_prompt(
    template_str: &str,
    messages: &[super::schema::ChatMessage],
) -> Result<String> {
    render_chat_prompt_with_tools(template_str, messages, None, false, None)
}

/// Context keys the renderer owns; a request whose `chat_template_kwargs`
/// names one of these is rejected before render (ADR-005 iter-229
/// Decision 4). `enable_thinking` is deliberately absent: kwargs may
/// override it (llama.cpp parity) — the merge order below makes kwargs
/// win every collision that survives this validation.
const RESERVED_TEMPLATE_KWARGS: &[&str] = &[
    "messages",
    "tools",
    "add_generation_prompt",
    "bos_token",
    "eos_token",
    "raise_exception",
];

/// Render the chat template with optional tool-definition exposure.
///
/// ADR-005 Phase 2a iter-133 Iter B production fix-forward: prior to this
/// iter, `tools` and per-message `tool_calls` / `tool_call_id` carried by
/// the request schema were silently dropped before render — every tool-aware
/// chat template (Gemma 4, Qwen 3.5/3.6, Llama 3.x) saw an empty `tools`
/// variable and emitted no tool-call definitions to the model. As a result,
/// the model never had a chance to invoke a tool even when the operator
/// declared one. This function threads them through:
///
///   1. `tools` (top-level Jinja variable): the raw OpenAI tool definitions,
///      serialized as JSON values. Templates check `{%- if tools -%}` before
///      iterating, so `None`/empty leaves existing behavior unchanged.
///   2. Per-message `tool_calls` (on assistant messages): each assistant
///      message in `messages` gets its `tool_calls` array exposed as a
///      Jinja-visible field. The template iterates and emits per-model markers
///      (e.g. Gemma 4's `<|tool_call>call:NAME{...}<tool_call|>`).
///   3. Synthetic `tool_responses` (on `role: "tool"` messages): OpenAI
///      represents tool results as `{role: "tool", tool_call_id, content}`
///      sibling messages. Most chat templates (Gemma 4 included) instead
///      expect a per-message `tool_responses: [{name, response}]` field.
///      We synthesize that field on each `role: "tool"` message by looking up
///      the function name from the prior assistant `tool_calls` keyed by
///      `tool_call_id`. The role itself is left verbatim — the template
///      decides whether to wrap with `<|turn>tool` or similar.
pub fn render_chat_prompt_with_tools(
    template_str: &str,
    messages: &[super::schema::ChatMessage],
    tools: Option<&[super::schema::Tool]>,
    enable_thinking: bool,
    chat_template_kwargs: Option<&serde_json::Map<String, serde_json::Value>>,
) -> Result<String> {
    use super::schema::MessageContent;

    // ADR-005 iter-229 Decision 4 step (a): reject renderer-owned keys
    // up front so the merge below can let kwargs win unconditionally.
    if let Some(kwargs) = chat_template_kwargs {
        for key in kwargs.keys() {
            if RESERVED_TEMPLATE_KWARGS.contains(&key.as_str()) {
                anyhow::bail!("reserved chat_template_kwargs key: {key}");
            }
        }
    }

    // DeepSeek-V4's published encoder is stateful: it merges consecutive
    // tool results into a user turn, sorts results by call order, and drops
    // old reasoning unless tools are active. Run the Rust behavioral port
    // instead of attempting to approximate those transitions through
    // minijinja. The GGUF still carries the Jinja form for external readers.
    if template_str == crate::core::chat_templates::DEEPSEEK_V4_FLASH_0731 {
        return render_deepseek_v4_prompt(messages, tools, enable_thinking, chat_template_kwargs);
    }

    let remap_assistant_to_model = template_str.contains("<|turn>model");

    // Lookup table tool_call_id → function-name for synthesizing
    // `tool_responses` on role:"tool" messages. Populated INCREMENTALLY
    // while walking messages (ADR-005 iter-229 Decision 5): a tool
    // message only resolves ids defined by PRIOR assistant turns, and a
    // duplicate id binds to the most recent prior definition. Forward
    // references and unknown ids fall back to "unknown".
    let mut id_to_name: std::collections::HashMap<String, String> =
        std::collections::HashMap::new();

    let mut out_msgs: Vec<serde_json::Value> = Vec::with_capacity(messages.len());
    for msg in messages {
        let mut role = msg.role.clone();
        if remap_assistant_to_model && role == "assistant" {
            role = "model".to_string();
        }
        let content_text = msg
            .content
            .as_ref()
            .map(|c| match c {
                MessageContent::Text(s) => s.clone(),
                MessageContent::Parts(_) => c.text(),
            })
            .unwrap_or_default();
        let mut obj = serde_json::Map::new();
        obj.insert("role".into(), serde_json::Value::String(role));
        obj.insert("content".into(), serde_json::Value::String(content_text));

        // Reasoning echo-back (ADR-005 iter-229 Decision 3): assistant
        // messages only. The Qwen 3.6 template's preserve branch
        // (qwen3-chatml.jinja:100) re-emits this as `<think>…</think>`
        // for tool-loop-tail turns; without it the model is shown a
        // fabricated empty think block as its own prior output.
        if msg.role == "assistant" {
            if let Some(rc) = msg.reasoning_content.as_ref() {
                obj.insert(
                    "reasoning_content".into(),
                    serde_json::Value::String(rc.clone()),
                );
            }
        }

        // Assistant tool_calls: serialize each as
        // `{id, type, function: {name, arguments}}`. `arguments` is the
        // raw OpenAI string UNLESS it parses to a JSON object, in which
        // case the mapping is substituted (ADR-005 iter-229 Decision 2)
        // — the Qwen template runs `tool_call.arguments|items`, which
        // requires a mapping. Arrays/scalars/parse failures keep the
        // string verbatim (today's shape).
        if let Some(tcs) = msg.tool_calls.as_ref() {
            let arr: Vec<serde_json::Value> = tcs
                .iter()
                .map(|tc| {
                    // Only ASSISTANT turns define ids (OpenAI semantics);
                    // tool_calls smuggled onto other roles must not poison
                    // the resolution map (gate-3 #1).
                    if msg.role == "assistant" {
                        id_to_name.insert(tc.id.clone(), tc.function.name.clone());
                    }
                    let arguments =
                        match serde_json::from_str::<serde_json::Value>(&tc.function.arguments) {
                            Ok(v @ serde_json::Value::Object(_)) => v,
                            _ => serde_json::Value::String(tc.function.arguments.clone()),
                        };
                    serde_json::json!({
                        "id": tc.id,
                        "type": tc.call_type,
                        "function": {
                            "name": tc.function.name,
                            "arguments": arguments,
                        },
                    })
                })
                .collect();
            obj.insert("tool_calls".into(), serde_json::Value::Array(arr));
        }

        // role:"tool" messages → synthesize `tool_responses` field. The
        // OpenAI shape is `{tool_call_id, content}`; we look up the tool
        // name from id_to_name and pass the content string verbatim.
        // Templates (Gemma 4) accept either string or mapping for the
        // `response` field; the string path is the safer default.
        if msg.role == "tool" {
            if let Some(tcid) = msg.tool_call_id.as_ref() {
                let name = id_to_name
                    .get(tcid)
                    .cloned()
                    .unwrap_or_else(|| "unknown".into());
                let response_str = msg
                    .content
                    .as_ref()
                    .map(|c| match c {
                        MessageContent::Text(s) => s.clone(),
                        MessageContent::Parts(_) => c.text(),
                    })
                    .unwrap_or_default();
                obj.insert(
                    "tool_responses".into(),
                    serde_json::json!([{"name": name, "response": response_str}]),
                );
            }
        }

        out_msgs.push(serde_json::Value::Object(obj));
    }

    // Tools serialize verbatim. Each Tool is `{type, function: {name,
    // description, parameters}}`; serde_json::to_value mirrors the wire
    // shape. Skipping the threading entirely when `None` keeps the existing
    // legacy callers (one-shot generate, overflow tokenize) byte-identical.
    let tools_json: serde_json::Value = match tools {
        None => serde_json::Value::Null,
        Some(t) if t.is_empty() => serde_json::Value::Null,
        Some(t) => serde_json::to_value(t).unwrap_or(serde_json::Value::Null),
    };

    // ADR-005 iter-229 Decision 1: shared env builder — the API path's
    // hand-rolled environment (pycompat only) diverged from the one-shot
    // renderer's (which had `tojson` + `raise_exception`), so every
    // `tools`-bearing request against the Qwen 3.6 template failed at
    // `tool | tojson` before inference. Strict raise policy: the
    // transcript is client-supplied, so template raise sites are
    // reachable and their message must surface in the 400 body.
    let mut env = crate::serve::build_chat_template_env(crate::serve::RaisePolicy::Strict);
    env.add_template("chat", template_str)
        .context("Failed to parse chat template as Jinja2")?;
    let tmpl = env
        .get_template("chat")
        .context("Failed to load parsed chat template")?;

    // Context assembly (ADR-005 iter-229 Decision 4): renderer values
    // first, then `chat_template_kwargs` merged OVER them — kwargs win
    // every collision that survived the reserved-key validation above
    // (i.e. only `enable_thinking`, the deliberate llama.cpp-parity
    // exception, plus free keys like `preserve_thinking`). Values pass
    // to Jinja verbatim; templates own their type checks.
    //
    // `enable_thinking` semantics (ADR-005 iter-133 Iter D, W67
    // unchanged): reasoning-capable templates branch on it to open or
    // suppress a thinking trace; templates that don't reference it are
    // unaffected.
    let mut ctx = serde_json::Map::new();
    ctx.insert("messages".into(), serde_json::Value::Array(out_msgs));
    ctx.insert("tools".into(), tools_json);
    ctx.insert(
        "enable_thinking".into(),
        serde_json::Value::Bool(enable_thinking),
    );
    ctx.insert(
        "add_generation_prompt".into(),
        serde_json::Value::Bool(true),
    );
    ctx.insert(
        "bos_token".into(),
        serde_json::Value::String("<bos>".into()),
    );
    ctx.insert(
        "eos_token".into(),
        serde_json::Value::String("<eos>".into()),
    );
    if let Some(kwargs) = chat_template_kwargs {
        for (k, v) in kwargs {
            ctx.insert(k.clone(), v.clone());
        }
    }

    let rendered = tmpl
        .render(minijinja::Value::from_serialize(&ctx))
        .context("Failed to render chat template")?;
    Ok(rendered)
}

fn render_deepseek_v4_prompt(
    messages: &[super::schema::ChatMessage],
    tools: Option<&[super::schema::Tool]>,
    enable_thinking: bool,
    kwargs: Option<&serde_json::Map<String, serde_json::Value>>,
) -> Result<String> {
    use crate::core::deepseek_v4_encoding::{
        encode_json, EncodeOptions, ReasoningEffort, ThinkingMode,
    };

    let thinking = kwargs
        .and_then(|v| v.get("thinking"))
        .or_else(|| kwargs.and_then(|v| v.get("enable_thinking")))
        .and_then(|v| v.as_bool())
        .unwrap_or(enable_thinking);
    let drop_thinking = kwargs
        .and_then(|v| v.get("drop_thinking"))
        .map(|v| {
            v.as_bool()
                .ok_or_else(|| anyhow::anyhow!("DeepSeek-V4 drop_thinking must be boolean"))
        })
        .transpose()?
        .unwrap_or(true);
    let reasoning_effort = match kwargs
        .and_then(|v| v.get("reasoning_effort"))
        .and_then(|v| v.as_str())
        .unwrap_or("low")
    {
        "low" => ReasoningEffort::Low,
        "high" => ReasoningEffort::High,
        "max" => ReasoningEffort::Max,
        other => {
            anyhow::bail!("DeepSeek-V4 reasoning_effort must be low, high, or max; got {other:?}")
        }
    };

    let mut values = serde_json::to_value(messages)
        .context("serialize DeepSeek-V4 messages")?
        .as_array()
        .cloned()
        .ok_or_else(|| anyhow::anyhow!("DeepSeek-V4 messages did not serialize as an array"))?;
    if let Some(tools) = tools.filter(|v| !v.is_empty()) {
        let tools = serde_json::to_value(tools).context("serialize DeepSeek-V4 tools")?;
        if let Some(target) = values.iter_mut().find(|v| {
            matches!(
                v.get("role").and_then(|r| r.as_str()),
                Some("system" | "developer")
            )
        }) {
            target
                .as_object_mut()
                .expect("serialized ChatMessage is an object")
                .insert("tools".into(), tools);
        } else {
            values.insert(
                0,
                serde_json::json!({"role": "system", "content": "", "tools": tools}),
            );
        }
    }

    let json = serde_json::to_string(&values).context("serialize DeepSeek-V4 transcript")?;
    encode_json(
        &json,
        EncodeOptions {
            thinking_mode: if thinking {
                ThinkingMode::Thinking
            } else {
                ThinkingMode::Chat
            },
            drop_thinking,
            add_bos: true,
            reasoning_effort,
        },
    )
    .map_err(anyhow::Error::from)
}

/// Resolve tokenizer path the same way `cmd_generate` does.
/// ADR-022 P1.11 — non-erroring sibling of `find_tokenizer` for the
/// GGUF-embedded path. Returns `Some(path)` only for (a) explicit
/// `--tokenizer <path>`, or (b) `tokenizer.json` next to the .gguf.
/// Returns `None` otherwise — the caller falls back to
/// `gemma4::tokenizer::build_tokenizer_from_gguf`.
fn resolve_tokenizer_path_optional(model_path: &Path, explicit: Option<&Path>) -> Option<PathBuf> {
    if let Some(p) = explicit {
        return Some(p.to_path_buf());
    }
    let dir = model_path.parent().unwrap_or(Path::new("."));
    let candidate = dir.join("tokenizer.json");
    if candidate.exists() {
        return Some(candidate);
    }
    None
}

#[allow(dead_code)]
fn find_tokenizer(model_path: &Path, explicit: Option<&Path>) -> Result<PathBuf> {
    // ADR-022 P1.8 / P1.10 — Same antipattern as the now-removed
    // `find_config` walk: previously walked `models/<subdir>/tokenizer.json`
    // and silently returned a peer model's tokenizer (e.g. qwen3.6's was
    // returned for a Gemma4 GGUF, producing token-id-mismatched garbage
    // output). Operator: "fallbacks in general are an antipattern" — the
    // walk is removed. Resolution order is now strict and explicit:
    //
    //   1. `--tokenizer <path>` (CLI flag)
    //   2. `tokenizer.json` next to the .gguf
    //
    // No filesystem walk fallback. Future P1.11 work: parse the embedded
    // tokenizer from GGUF metadata (`tokenizer.ggml.tokens`, scores,
    // merges, special-token ids) so the on-disk tokenizer.json is no
    // longer required either, mirroring how llama.cpp self-bootstraps.
    if let Some(p) = explicit {
        return Ok(p.to_path_buf());
    }
    let dir = model_path.parent().unwrap_or(Path::new("."));
    let candidate = dir.join("tokenizer.json");
    if candidate.exists() {
        return Ok(candidate);
    }
    anyhow::bail!(
        "Cannot find tokenizer.json next to {}. Use --tokenizer <path> to specify it explicitly. \
         (The legacy `models/<subdir>/tokenizer.json` walk was removed under ADR-022 P1.10 — it \
         was silently picking peer models' tokenizers.)",
        model_path.display()
    )
}

/// Resolve config.json path (same heuristics as cmd_generate).
#[allow(dead_code)]
fn find_config(model_path: &Path, explicit: Option<&Path>) -> Result<PathBuf> {
    if let Some(p) = explicit {
        return Ok(p.to_path_buf());
    }
    let dir = model_path.parent().unwrap_or(Path::new("."));
    let candidate = dir.join("config.json");
    if candidate.exists() {
        return Ok(candidate);
    }
    for subdir in &["gemma4", "gemma-4"] {
        let candidate = Path::new("models").join(subdir).join("config.json");
        if candidate.exists() {
            return Ok(candidate);
        }
    }
    let models_dir = Path::new("models");
    if models_dir.is_dir() {
        for entry in std::fs::read_dir(models_dir)? {
            let entry = entry?;
            if entry.path().is_dir() {
                let c = entry.path().join("config.json");
                if c.exists() {
                    return Ok(c);
                }
            }
        }
    }
    anyhow::bail!("Cannot find config.json. Use --config to specify the path explicitly.")
}

// ---------------------------------------------------------------------------
// Tests
// ---------------------------------------------------------------------------

#[cfg(test)]
mod tests {
    use super::super::schema::{ChatMessage, ContentPart, ImageUrl, MessageContent};
    use super::*;

    #[test]
    fn sampling_params_default_is_greedy_t0() {
        let p = SamplingParams::default();
        assert_eq!(p.temperature, 0.0);
        assert_eq!(p.top_p, 1.0);
        assert_eq!(p.top_k, 0);
        assert_eq!(p.repetition_penalty, 1.0);
        assert_eq!(p.max_tokens, 512);
        assert!(p.stop_strings.is_empty());
    }

    #[test]
    fn hit_stop_string_empty_stops_is_false() {
        assert!(!hit_stop_string("anything", &[]));
    }

    // -----------------------------------------------------------------
    // "gemma-hybrid-lcp" (2026-08-03) — build_gemma_lcp_payload
    // -----------------------------------------------------------------

    fn lcp_test_dense_arc(
        dev: &mlx_native::MlxDevice,
        cap: usize,
    ) -> std::sync::Arc<crate::inference::models::gemma4::DenseKvBuffers> {
        std::sync::Arc::new(crate::inference::models::gemma4::DenseKvBuffers {
            k: dev
                .alloc_buffer(2 * cap * 4 * 4, mlx_native::DType::F32, vec![2, cap, 4])
                .unwrap(),
            v: dev
                .alloc_buffer(2 * cap * 4 * 4, mlx_native::DType::F32, vec![2, cap, 4])
                .unwrap(),
            capacity: cap,
            is_sliding: false,
            dtype: mlx_native::DType::F32,
        })
    }

    fn lcp_test_hybrid_arc(
        dev: &mlx_native::MlxDevice,
        cap: usize,
    ) -> std::sync::Arc<crate::inference::models::gemma4::HybridKvBuffers> {
        std::sync::Arc::new(crate::inference::models::gemma4::HybridKvBuffers {
            k: dev
                .alloc_buffer(2 * cap * 4 * 2, mlx_native::DType::F16, vec![2, cap, 4])
                .unwrap(),
            v_packed: dev
                .alloc_buffer(2 * cap * 4, mlx_native::DType::U8, vec![2, cap, 4])
                .unwrap(),
            v_norms: dev
                .alloc_buffer(2 * cap * 4, mlx_native::DType::F32, vec![2, cap])
                .unwrap(),
            capacity: cap,
            is_sliding: false,
            norms_per_pos: 1,
            bf16_xlen_k: None,
            bf16_xlen_v: None,
        })
    }

    #[test]
    fn build_gemma_lcp_payload_dense_only_yields_dense_variants() {
        let _gpu = crate::inference::hf2q_gpu_test_lock();
        let dev = mlx_native::MlxDevice::new().expect("device");
        let dense = vec![lcp_test_dense_arc(&dev, 8), lcp_test_dense_arc(&dev, 8)];
        let payload = build_gemma_lcp_payload(dense, None).expect("payload");
        assert_eq!(payload.len(), 2);
        for arc in &payload {
            assert!(
                arc.hybrid().is_none(),
                "dense-only regime must not carry hybrid legs"
            );
            assert_eq!(arc.dense().capacity, 8);
        }
    }

    #[test]
    fn build_gemma_lcp_payload_zips_legs_and_preserves_contents() {
        let _gpu = crate::inference::hf2q_gpu_test_lock();
        let dev = mlx_native::MlxDevice::new().expect("device");
        let mut d = lcp_test_dense_arc(&dev, 8);
        // Plant a canary in the dense K buffer.
        {
            let mut owned =
                std::sync::Arc::try_unwrap(d).unwrap_or_else(|_| panic!("exclusive arc"));
            owned.k.as_mut_slice::<u8>().unwrap()[0] = 0xAB;
            d = std::sync::Arc::new(owned);
        }
        let mut h = lcp_test_hybrid_arc(&dev, 8);
        {
            let mut owned =
                std::sync::Arc::try_unwrap(h).unwrap_or_else(|_| panic!("exclusive arc"));
            owned.v_packed.as_mut_slice::<u8>().unwrap()[0] = 0xCD;
            h = std::sync::Arc::new(owned);
        }
        let payload = build_gemma_lcp_payload(vec![d], Some(vec![h])).expect("payload");
        assert_eq!(payload.len(), 1);
        let layer = &payload[0];
        let h_leg = layer
            .hybrid()
            .expect("hybrid leg present under hybrid regime");
        assert_eq!(
            layer.dense().k.as_slice::<u8>().unwrap()[0],
            0xAB,
            "dense leg contents must ride through"
        );
        assert_eq!(
            h_leg.v_packed.as_slice::<u8>().unwrap()[0],
            0xCD,
            "hybrid leg contents must ride through"
        );
    }

    #[test]
    fn build_gemma_lcp_payload_layer_mismatch_fails_safe() {
        let _gpu = crate::inference::hf2q_gpu_test_lock();
        let dev = mlx_native::MlxDevice::new().expect("device");
        let dense = vec![lcp_test_dense_arc(&dev, 8), lcp_test_dense_arc(&dev, 8)];
        let hybrid = vec![lcp_test_hybrid_arc(&dev, 8)];
        assert!(
            build_gemma_lcp_payload(dense, Some(hybrid)).is_none(),
            "layer-count mismatch must skip the store (never pair wrong legs)"
        );
    }

    #[test]
    fn hit_stop_string_matches_trailing() {
        let stops = vec!["END".to_string()];
        assert!(hit_stop_string("blah END", &stops));
        assert!(!hit_stop_string("END blah", &stops));
        assert!(!hit_stop_string("blah", &stops));
    }

    #[test]
    fn hit_stop_string_ignores_empty_stop() {
        let stops = vec!["".to_string(), "END".to_string()];
        // Empty strings should not cause false positives.
        assert!(!hit_stop_string("blah", &stops));
        assert!(hit_stop_string("blah END", &stops));
    }

    #[test]
    fn strip_trailing_stop_removes_suffix() {
        let mut s = String::from("hello END");
        strip_trailing_stop(&mut s, &["END".to_string()]);
        assert_eq!(s, "hello ");
    }

    #[test]
    fn strip_trailing_stop_no_match_leaves_unchanged() {
        let mut s = String::from("hello");
        strip_trailing_stop(&mut s, &["END".to_string()]);
        assert_eq!(s, "hello");
    }

    #[test]
    fn render_chat_prompt_single_user_round_trip() {
        // A minimal Jinja template that just formats role:content per line.
        let tmpl = r#"{%- for m in messages -%}
{{ m.role }}: {{ m.content }}
{%- endfor -%}
{%- if add_generation_prompt -%}
assistant:
{%- endif -%}"#;
        let msgs = vec![ChatMessage {
            role: "user".into(),
            content: Some(MessageContent::Text("hi".into())),
            reasoning_content: None,
            tool_calls: None,
            tool_call_id: None,
            name: None,
        }];
        let out = render_chat_prompt(tmpl, &msgs).unwrap();
        assert!(out.contains("user: hi"));
        assert!(out.ends_with("assistant:"));
    }

    #[test]
    fn render_chat_prompt_remaps_assistant_for_gemma_template() {
        // Template that contains the Gemma 4 marker `<|turn>model` triggers
        // the assistant→model remap.
        let tmpl = "<|turn>system\n<|turn>user\n{% for m in messages %}{{ m.role }}:{{ m.content }}\n{% endfor %}<|turn>model\n";
        let msgs = vec![
            ChatMessage {
                role: "user".into(),
                content: Some(MessageContent::Text("hi".into())),
                reasoning_content: None,
                tool_calls: None,
                tool_call_id: None,
                name: None,
            },
            ChatMessage {
                role: "assistant".into(),
                content: Some(MessageContent::Text("hello".into())),
                reasoning_content: None,
                tool_calls: None,
                tool_call_id: None,
                name: None,
            },
        ];
        let out = render_chat_prompt(tmpl, &msgs).unwrap();
        assert!(out.contains("user:hi"));
        // assistant should have been remapped to model
        assert!(out.contains("model:hello"));
        assert!(!out.contains("assistant:hello"));
    }

    #[test]
    fn fallback_gemma4_api_template_appends_empty_channel_block() {
        // iter-217 regression guard. The upstream Gemma 4 chat template
        // (vllm/examples/tool_chat_template_gemma4.jinja:326-330) appends
        // `<|channel>thought\n<channel|>` after `<|turn>model\n` when
        // `enable_thinking=false` (the default). Without this empty-block,
        // the model emits a stray `<channel|>` close marker as its first
        // decoded token (training expects channel closed before content).
        // The ReasoningSplitter requires both open + close in OUTPUT to
        // extract reasoning; a lone close has no open to match and leaks
        // verbatim into delta.content. This test pins the prompt-side
        // empty-block so the regression is caught at build time, not by
        // an operator inspecting curl bytes.
        use crate::serve::FALLBACK_GEMMA4_API_CHAT_TEMPLATE;
        assert!(
            FALLBACK_GEMMA4_API_CHAT_TEMPLATE
                .ends_with("<|turn>model\n<|channel>thought\n<channel|>"),
            "FALLBACK_GEMMA4_API_CHAT_TEMPLATE must end with the empty channel block; \
             got tail: {:?}",
            &FALLBACK_GEMMA4_API_CHAT_TEMPLATE
                [FALLBACK_GEMMA4_API_CHAT_TEMPLATE.len().saturating_sub(50)..]
        );
    }

    #[test]
    fn fallback_gemma4_api_template_renders_with_empty_channel_block_terminator() {
        // End-to-end render of the API fallback template: a single user
        // message must produce a prompt that ends with the empty channel
        // block, so the model's first decoded token continues AFTER the
        // close marker (no leak).
        use crate::serve::FALLBACK_GEMMA4_API_CHAT_TEMPLATE;
        let msgs = vec![ChatMessage {
            role: "user".into(),
            content: Some(MessageContent::Text("hi".into())),
            reasoning_content: None,
            tool_calls: None,
            tool_call_id: None,
            name: None,
        }];
        let out = render_chat_prompt(FALLBACK_GEMMA4_API_CHAT_TEMPLATE, &msgs).unwrap();
        assert!(
            out.ends_with("<|channel>thought\n<channel|>"),
            "rendered Gemma4 fallback prompt must terminate with empty channel block; \
             got tail: {:?}",
            &out[out.len().saturating_sub(50)..]
        );
        // Sanity: the empty block lives after `<|turn>model\n`, not before.
        let model_turn_idx = out
            .find("<|turn>model\n")
            .expect("`<|turn>model` marker present");
        let channel_open_idx = out.find("<|channel>").expect("`<|channel>` open present");
        assert!(
            channel_open_idx > model_turn_idx,
            "`<|channel>` must come AFTER `<|turn>model\\n` in the rendered prompt; \
             got channel_open_idx={channel_open_idx} model_turn_idx={model_turn_idx}"
        );
    }

    #[test]
    fn render_chat_prompt_does_not_remap_for_non_gemma_template() {
        let tmpl = "{% for m in messages %}{{ m.role }}:{{ m.content }}\n{% endfor %}";
        let msgs = vec![ChatMessage {
            role: "assistant".into(),
            content: Some(MessageContent::Text("hello".into())),
            reasoning_content: None,
            tool_calls: None,
            tool_call_id: None,
            name: None,
        }];
        let out = render_chat_prompt(tmpl, &msgs).unwrap();
        assert!(out.contains("assistant:hello"));
    }

    #[test]
    fn render_chat_prompt_handles_pythonic_string_methods_via_pycompat() {
        // ADR-005 Phase 2a iter-133 Iter A regression test for the
        // pycompat side-fix in `render_chat_prompt`. Real-world chat
        // templates (Gemma 4's `strip_thinking` macro is the surfaced
        // case) call Python-string methods like `.split()` directly on
        // string values; minijinja's vanilla Environment doesn't expose
        // those, so a multi-turn render that exercises the macro fails
        // with `UnknownMethod: string has no method named split` at the
        // second user turn.
        //
        // This template is a minimal stand-in: a `{%- macro -%}` that
        // calls `text.split('|')` (Python-style), invoked once per
        // assistant message inside a `for messages` loop. Without the
        // pycompat callback this `render_chat_prompt` call panics on
        // unwrap; with the callback it renders cleanly.
        let tmpl = "<|turn>model\n\
                    {%- macro splitter(text) -%}\
                    {%- for part in text.split('|') -%}{{ part }}+{% endfor -%}\
                    {%- endmacro -%}\
                    {% for m in messages %}\
                    {%- if m.role == 'model' -%}M:{{ splitter(m.content) }}\n\
                    {%- else -%}{{ m.role }}:{{ m.content }}\n{% endif -%}\
                    {% endfor %}";
        let msgs = vec![
            ChatMessage {
                role: "user".into(),
                content: Some(MessageContent::Text("hi".into())),
                reasoning_content: None,
                tool_calls: None,
                tool_call_id: None,
                name: None,
            },
            ChatMessage {
                role: "assistant".into(),
                content: Some(MessageContent::Text("a|b|c".into())),
                reasoning_content: None,
                tool_calls: None,
                tool_call_id: None,
                name: None,
            },
            ChatMessage {
                role: "user".into(),
                content: Some(MessageContent::Text("again".into())),
                reasoning_content: None,
                tool_calls: None,
                tool_call_id: None,
                name: None,
            },
        ];
        let out = render_chat_prompt(tmpl, &msgs).unwrap();
        assert!(out.contains("user:hi"), "out={out}");
        // assistant remapped to model (gemma marker present), then split('|')
        // produced ["a", "b", "c"], each with a trailing '+'.
        assert!(out.contains("M:a+b+c+"), "out={out}");
        assert!(out.contains("user:again"), "out={out}");
    }

    #[test]
    fn render_chat_prompt_with_tools_threads_tools_into_jinja_context() {
        // ADR-005 Phase 2a iter-133 Iter B fix-forward: prior to this iter,
        // `tools` declared on a chat-completions request were silently
        // dropped before render — every tool-aware chat template (Gemma 4,
        // Qwen 3.5/3.6, Llama 3.x) saw an empty `tools` variable and emitted
        // no tool definitions to the model. Regression test: a minimal
        // template that just emits "TOOLS:<count>\n" + tool names
        // round-trips correctly.
        let tmpl = "{%- if tools -%}TOOLS:{{ tools | length }}\n\
                    {%- for t in tools -%}{{ t.function.name }};{%- endfor -%}\n\
                    {%- endif -%}\
                    {%- for m in messages -%}{{ m.role }}:{{ m.content }}\n{%- endfor -%}";
        let msgs = vec![ChatMessage {
            role: "user".into(),
            content: Some(MessageContent::Text("weather?".into())),
            reasoning_content: None,
            tool_calls: None,
            tool_call_id: None,
            name: None,
        }];
        let tools = vec![
            super::super::schema::Tool {
                tool_type: "function".into(),
                function: super::super::schema::ToolFunction {
                    name: "get_current_weather".into(),
                    description: Some("Look up the weather".into()),
                    parameters: Some(serde_json::json!({
                        "type": "object",
                        "properties": {
                            "location": {"type": "string"}
                        },
                        "required": ["location"]
                    })),
                },
            },
            super::super::schema::Tool {
                tool_type: "function".into(),
                function: super::super::schema::ToolFunction {
                    name: "get_news".into(),
                    description: None,
                    parameters: None,
                },
            },
        ];
        let out = render_chat_prompt_with_tools(tmpl, &msgs, Some(&tools), false, None).unwrap();
        assert!(out.contains("TOOLS:2"), "out={out}");
        assert!(out.contains("get_current_weather;"), "out={out}");
        assert!(out.contains("get_news;"), "out={out}");
        assert!(out.contains("user:weather?"), "out={out}");

        // None / empty path → tools block must NOT fire.
        let out_none = render_chat_prompt_with_tools(tmpl, &msgs, None, false, None).unwrap();
        assert!(!out_none.contains("TOOLS:"), "out_none={out_none}");
        let out_empty = render_chat_prompt_with_tools(tmpl, &msgs, Some(&[]), false, None).unwrap();
        assert!(!out_empty.contains("TOOLS:"), "out_empty={out_empty}");

        // Legacy entry-point `render_chat_prompt` (no tools param) should be
        // byte-identical to the empty/None path.
        let out_legacy = render_chat_prompt(tmpl, &msgs).unwrap();
        assert_eq!(out_legacy, out_none);
    }

    #[test]
    fn deepseek_v4_template_dispatches_to_rust_encoder() {
        let msgs = vec![
            ChatMessage {
                role: "system".into(),
                content: Some(MessageContent::Text("Be exact.".into())),
                reasoning_content: None,
                tool_calls: None,
                tool_call_id: None,
                name: None,
            },
            ChatMessage {
                role: "user".into(),
                content: Some(MessageContent::Text("Weather?".into())),
                reasoning_content: None,
                tool_calls: None,
                tool_call_id: None,
                name: None,
            },
        ];
        let tools = vec![super::super::schema::Tool {
            tool_type: "function".into(),
            function: super::super::schema::ToolFunction {
                name: "weather".into(),
                description: Some("Get weather".into()),
                parameters: Some(serde_json::json!({
                    "type": "object",
                    "properties": {"city": {"type": "string"}}
                })),
            },
        }];

        let out = render_chat_prompt_with_tools(
            crate::core::chat_templates::DEEPSEEK_V4_FLASH_0731,
            &msgs,
            Some(&tools),
            false,
            None,
        )
        .unwrap();
        assert!(out.starts_with("<｜begin▁of▁sentence｜>Be exact.\n\n## Tools"));
        assert!(out.contains("\"name\": \"weather\""));
        assert!(out.ends_with("<｜User｜>Weather?<｜Assistant｜></think>"));
    }

    #[test]
    fn render_chat_prompt_with_tools_threads_per_message_tool_calls_and_responses() {
        // Verify the per-message threading of `tool_calls` (assistant) and
        // synthesized `tool_responses` (from role:"tool" messages, looked up
        // by tool_call_id). A minimal template iterates messages and emits
        // tool_calls + tool_responses verbatim.
        let tmpl = "{%- for m in messages -%}\
                    [{{ m.role }}]\
                    {%- if m.tool_calls -%}\
                    {%- for tc in m.tool_calls -%}TC:{{ tc.function.name }}({{ tc.function.arguments }});{%- endfor -%}\
                    {%- endif -%}\
                    {%- if m.tool_responses -%}\
                    {%- for tr in m.tool_responses -%}TR:{{ tr.name }}={{ tr.response }};{%- endfor -%}\
                    {%- endif -%}\
                    {%- if m.content -%}{{ m.content }}{%- endif -%}\n\
                    {%- endfor -%}";
        let msgs = vec![
            ChatMessage {
                role: "user".into(),
                content: Some(MessageContent::Text("Paris weather?".into())),
                reasoning_content: None,
                tool_calls: None,
                tool_call_id: None,
                name: None,
            },
            ChatMessage {
                role: "assistant".into(),
                content: None,
                reasoning_content: None,
                tool_calls: Some(vec![super::super::schema::ToolCall {
                    id: "call_abc".into(),
                    call_type: "function".into(),
                    function: super::super::schema::ToolCallFunction {
                        name: "get_current_weather".into(),
                        // Deliberately NOT valid JSON: iter-229 Decision 2
                        // substitutes a mapping for valid object strings, so
                        // this test keeps a malformed string to pin the
                        // verbatim-threading path; shape coverage lives in
                        // the iter-229 AC2 tests below.
                        arguments: "location=Paris".into(),
                    },
                }]),
                tool_call_id: None,
                name: None,
            },
            ChatMessage {
                role: "tool".into(),
                content: Some(MessageContent::Text("{\"temperature\": 18}".into())),
                reasoning_content: None,
                tool_calls: None,
                tool_call_id: Some("call_abc".into()),
                name: None,
            },
        ];
        let out = render_chat_prompt_with_tools(tmpl, &msgs, None, false, None).unwrap();
        assert!(out.contains("[user]Paris weather?"), "out={out}");
        // Assistant message: tool_calls visible verbatim (non-object
        // arguments string is preserved raw per iter-229 Decision 2).
        assert!(
            out.contains("[assistant]TC:get_current_weather(location=Paris);"),
            "out={out}"
        );
        // Tool message: tool_responses synthesized via id_to_name lookup.
        assert!(
            out.contains("[tool]TR:get_current_weather={\"temperature\": 18};"),
            "out={out}"
        );
    }

    #[test]
    fn render_chat_prompt_concatenates_multimodal_text_parts() {
        let tmpl = "{% for m in messages %}{{ m.content }}|{% endfor %}";
        let msgs = vec![ChatMessage {
            role: "user".into(),
            content: Some(MessageContent::Parts(vec![
                ContentPart::Text {
                    text: "what is ".into(),
                },
                ContentPart::ImageUrl {
                    image_url: ImageUrl {
                        url: "data:image/png;base64,XXX".into(),
                        detail: None,
                    },
                },
                ContentPart::Text {
                    text: "this?".into(),
                },
            ])),
            reasoning_content: None,
            tool_calls: None,
            tool_call_id: None,
            name: None,
        }];
        let out = render_chat_prompt(tmpl, &msgs).unwrap();
        // Image part is silently dropped (iter 3 scope); text parts joined.
        assert_eq!(out.trim(), "what is this?|");
    }

    // ---- ADR-005 iter-229: env parity + tool-args mapping + reasoning ----
    //
    // Spike-driven ACs; the vendor qwen3-chatml fixture is byte-identical
    // to the GGUF-embedded template of the served Qwen 3.6 model, so these
    // tests exercise the REAL serve-path render.

    fn i229_msg(role: &str, content: &str) -> ChatMessage {
        ChatMessage {
            role: role.into(),
            content: Some(MessageContent::Text(content.into())),
            reasoning_content: None,
            tool_calls: None,
            tool_call_id: None,
            name: None,
        }
    }

    fn i229_tool_call(id: &str, name: &str, arguments: &str) -> super::super::schema::ToolCall {
        super::super::schema::ToolCall {
            id: id.into(),
            call_type: "function".into(),
            function: super::super::schema::ToolCallFunction {
                name: name.into(),
                arguments: arguments.into(),
            },
        }
    }

    fn i229_tools() -> Vec<super::super::schema::Tool> {
        vec![super::super::schema::Tool {
            tool_type: "function".into(),
            function: super::super::schema::ToolFunction {
                name: "read_file".into(),
                description: Some("Read a file".into()),
                parameters: Some(serde_json::json!(
                    {"type": "object", "properties": {"path": {"type": "string"}}}
                )),
            },
        }]
    }

    /// Agentic tool-loop-tail transcript: system, user, assistant tool
    /// call (arguments as the given string), tool result.
    fn i229_tail_transcript(arguments: &str) -> Vec<ChatMessage> {
        let mut assistant = i229_msg("assistant", "");
        assistant.tool_calls = Some(vec![i229_tool_call("c1", "read_file", arguments)]);
        let mut tool = i229_msg("tool", "fn main() { panic!() }");
        tool.tool_call_id = Some("c1".into());
        vec![
            i229_msg("system", "You are an agent."),
            i229_msg("user", "Fix the bug in foo.rs"),
            assistant,
            tool,
        ]
    }

    #[test]
    fn iter229_ac1_tools_render_via_vendor_qwen_template() {
        // Pre-fix: `unknown filter: tojson` at template line 50 — every
        // tools-bearing request against the served qwen3.6 died here.
        let msgs = vec![
            i229_msg("system", "You are an agent."),
            i229_msg("user", "Fix the bug in foo.rs"),
        ];
        let out = render_chat_prompt_with_tools(
            crate::core::chat_templates::QWEN3_CHATML,
            &msgs,
            Some(&i229_tools()),
            true,
            None,
        )
        .unwrap();
        assert!(out.contains("<tools>"), "out={out}");
        assert!(out.contains("\"read_file\""), "out={out}");
    }

    #[test]
    fn iter229_ac2a_object_arguments_render_as_mapping() {
        // Pre-fix: string arguments hit `arguments|items` (template line
        // 120) → `cannot convert value into pairs` on any echo-back.
        let out = render_chat_prompt_with_tools(
            crate::core::chat_templates::QWEN3_CHATML,
            &i229_tail_transcript("{\"path\": \"foo.rs\"}"),
            Some(&i229_tools()),
            true,
            None,
        )
        .unwrap();
        assert!(out.contains("<function=read_file>"), "out={out}");
        assert!(
            out.contains("<parameter=path>\nfoo.rs\n</parameter>"),
            "out={out}"
        );
    }

    #[test]
    fn iter229_ac2bcd_non_object_arguments_stay_raw_strings() {
        // Raw-preservation semantics observed via an inspection template
        // (the vendor template can't render non-object arguments — pinned
        // separately below).
        let tmpl = "{{ messages[2].tool_calls[0].function.arguments is string }}";
        for raw in ["[1,2]", "42", "true", "null", "\"foo\"", "location=Paris"] {
            let out =
                render_chat_prompt_with_tools(tmpl, &i229_tail_transcript(raw), None, false, None)
                    .unwrap();
            assert_eq!(out, "true", "arguments {raw:?} must stay a raw string");
        }
        // Control: a valid object DOES become a mapping.
        let out = render_chat_prompt_with_tools(
            tmpl,
            &i229_tail_transcript("{\"path\": \"foo.rs\"}"),
            None,
            false,
            None,
        )
        .unwrap();
        assert_eq!(out, "false", "object arguments must become a mapping");
    }

    #[test]
    fn iter229_ac2_pin_vendor_template_fails_clean_on_non_object_arguments() {
        // Non-spec arguments (not a JSON object) still can't render
        // through the qwen template's `|items` — unchanged from today,
        // now documented. The error must be a clean Err (mapped to 400
        // by the handler), not a panic. All six preserved shapes.
        for raw in ["[1,2]", "42", "true", "null", "\"foo\"", "location=Paris"] {
            let err = render_chat_prompt_with_tools(
                crate::core::chat_templates::QWEN3_CHATML,
                &i229_tail_transcript(raw),
                Some(&i229_tools()),
                true,
                None,
            )
            .unwrap_err();
            assert!(
                format!("{err:#}").contains("Failed to render chat template"),
                "arguments {raw:?}: err={err:#}"
            );
        }
    }

    /// The REAL Gemma 4 chat template as served: extracted verbatim from
    /// `/opt/hf2q/models/gemma4/gemma4-ara-2pass-APEX-Q5_K_M.gguf`
    /// metadata `tokenizer.chat_template` (12045 bytes, 2026-07-09,
    /// ADR-005 iter-229 gate-3 #3). Handles BOTH argument shapes natively
    /// (`function['arguments'] is mapping` → dictsort iteration at
    /// template line 193; `is string` → verbatim at line 200).
    const GEMMA4_EMBEDDED_TEMPLATE: &str =
        include_str!("test_fixtures/gemma4-apex-embedded-chat-template.jinja");

    #[test]
    fn iter229_ac2e_gemma_dual_shape_arguments_byte_pins() {
        // Object-string arguments → mapping render (dictsort path).
        let obj = render_chat_prompt_with_tools(
            GEMMA4_EMBEDDED_TEMPLATE,
            &i229_tail_transcript("{\"path\": \"foo.rs\"}"),
            None,
            false,
            None,
        )
        .unwrap();
        assert_eq!(
            obj,
            "<bos><|turn>system\nYou are an agent.<turn|>\n<|turn>user\nFix the bug in foo.rs<turn|>\n<|turn>model\n<|tool_call>call:read_file{path:<|\"|>foo.rs<|\"|>}<tool_call|><turn|>\n<|turn>tool\n<|tool_response>response:read_file{value:<|\"|>fn main() { panic!() }<|\"|>}<tool_response|>fn main() { panic!() }<turn|>\n<|channel>thought\n<channel|>"
        );
        // Non-object arguments → raw string verbatim (template's own
        // `is string` branch).
        let raw = render_chat_prompt_with_tools(
            GEMMA4_EMBEDDED_TEMPLATE,
            &i229_tail_transcript("location=Paris"),
            None,
            false,
            None,
        )
        .unwrap();
        assert_eq!(
            raw,
            "<bos><|turn>system\nYou are an agent.<turn|>\n<|turn>user\nFix the bug in foo.rs<turn|>\n<|turn>model\n<|tool_call>call:read_file{location=Paris}<tool_call|><turn|>\n<|turn>tool\n<|tool_response>response:read_file{value:<|\"|>fn main() { panic!() }<|\"|>}<tool_response|>fn main() { panic!() }<turn|>\n<|channel>thought\n<channel|>"
        );
    }

    #[test]
    fn iter229_ac4b_gemma_absent_kwargs_golden() {
        // Gemma half of AC4b: no-tools/no-reasoning/no-kwargs transcript
        // byte-pinned through the REAL embedded template.
        let msgs = vec![
            i229_msg("system", "You are an agent."),
            i229_msg("user", "Fix the bug in foo.rs"),
            i229_msg("assistant", "Done, fixed the panic."),
            i229_msg("user", "Now add a test for it"),
        ];
        let out = render_chat_prompt_with_tools(GEMMA4_EMBEDDED_TEMPLATE, &msgs, None, true, None)
            .unwrap();
        assert_eq!(
            out,
            "<bos><|turn>system\n<|think|>You are an agent.<turn|>\n<|turn>user\nFix the bug in foo.rs<turn|>\n<|turn>model\nDone, fixed the panic.<turn|>\n<|turn>user\nNow add a test for it<turn|>\n<|turn>model\n"
        );
        let empty = serde_json::Map::new();
        let out_empty = render_chat_prompt_with_tools(
            GEMMA4_EMBEDDED_TEMPLATE,
            &msgs,
            None,
            true,
            Some(&empty),
        )
        .unwrap();
        assert_eq!(out, out_empty);
    }

    #[test]
    fn iter229_gate3_non_assistant_tool_calls_do_not_define_ids() {
        // tool_calls smuggled onto a user message must not populate the
        // id→name map (gate-3 #1): the later tool message stays "unknown".
        let inspect = "{% for m in messages %}{% if m.tool_responses %}\
                       {% for tr in m.tool_responses %}TR:{{ tr.name }};{% endfor %}\
                       {% endif %}{% endfor %}";
        let mut smuggler = i229_msg("user", "go");
        smuggler.tool_calls = Some(vec![i229_tool_call("P", "poisoned_fn", "{}")]);
        let mut t = i229_msg("tool", "r");
        t.tool_call_id = Some("P".into());
        let msgs = vec![smuggler, i229_msg("user", "really go"), t];
        let out = render_chat_prompt_with_tools(inspect, &msgs, None, false, None).unwrap();
        assert_eq!(out, "TR:unknown;");
    }

    #[test]
    fn iter229_ac3a_reasoning_content_replayed_on_tail_turn() {
        let mut msgs = i229_tail_transcript("{\"path\": \"foo.rs\"}");
        msgs[2].reasoning_content = Some("I should read the file first.".into());
        let out = render_chat_prompt_with_tools(
            crate::core::chat_templates::QWEN3_CHATML,
            &msgs,
            Some(&i229_tools()),
            true,
            None,
        )
        .unwrap();
        assert!(
            out.contains("<think>\nI should read the file first.\n</think>"),
            "tail-turn reasoning must be replayed verbatim; out={out}"
        );
    }

    #[test]
    fn iter229_ac3b_absent_reasoning_renders_empty_think_block() {
        // Today's template behavior for a reasoning-less tail turn,
        // byte-pinned in full: the preserve branch fires
        // (index > last_query_index) with an empty reasoning_content —
        // the model is shown `<think>\n\n</think>` as its prior output.
        let out = render_chat_prompt_with_tools(
            crate::core::chat_templates::QWEN3_CHATML,
            &i229_tail_transcript("{\"path\": \"foo.rs\"}"),
            Some(&i229_tools()),
            true,
            None,
        )
        .unwrap();
        assert_eq!(
            out,
            "<|im_start|>system\n# Tools\n\nYou have access to the following functions:\n\n<tools>\n{\"function\":{\"description\":\"Read a file\",\"name\":\"read_file\",\"parameters\":{\"properties\":{\"path\":{\"type\":\"string\"}},\"type\":\"object\"}},\"type\":\"function\"}\n</tools>\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>\n\nYou are an agent.<|im_end|>\n<|im_start|>user\nFix the bug in foo.rs<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n<tool_call>\n<function=read_file>\n<parameter=path>\nfoo.rs\n</parameter>\n</function>\n</tool_call><|im_end|>\n<|im_start|>user\n<tool_response>\nfn main() { panic!() }\n</tool_response><|im_end|>\n<|im_start|>assistant\n<think>\n"
        );
    }

    #[test]
    fn iter229_ac3c_reasoning_only_inserted_on_assistant_messages() {
        let tmpl =
            "{% for m in messages %}{{ m.role }}:{{ m.reasoning_content is defined }};{% endfor %}";
        let mut msgs = vec![
            i229_msg("system", "s"),
            i229_msg("user", "u"),
            i229_msg("assistant", "a"),
            i229_msg("tool", "t"),
        ];
        for m in msgs.iter_mut() {
            m.reasoning_content = Some("leak?".into());
        }
        msgs[3].tool_call_id = Some("c1".into());
        let out = render_chat_prompt_with_tools(tmpl, &msgs, None, false, None).unwrap();
        assert_eq!(
            out, "system:false;user:false;assistant:true;tool:false;",
            "reasoning_content must reach the context on assistant messages only"
        );
    }

    #[test]
    fn iter229_ac4a_preserve_thinking_kwarg_replays_pre_query_reasoning() {
        let mut a1 = i229_msg("assistant", "Done, fixed the panic.");
        a1.reasoning_content = Some("Turn-1 reasoning: the panic was a stub.".into());
        let msgs = vec![
            i229_msg("system", "You are an agent."),
            i229_msg("user", "Fix the bug in foo.rs"),
            a1,
            i229_msg("user", "Now add a test for it"),
        ];
        let mut kwargs = serde_json::Map::new();
        kwargs.insert("preserve_thinking".into(), serde_json::Value::Bool(true));
        let with = render_chat_prompt_with_tools(
            crate::core::chat_templates::QWEN3_CHATML,
            &msgs,
            None,
            true,
            Some(&kwargs),
        )
        .unwrap();
        assert!(
            with.contains("<think>\nTurn-1 reasoning: the panic was a stub.\n</think>"),
            "with={with}"
        );
        // Without the kwarg: stripped per Qwen convention (spike C1).
        let without = render_chat_prompt_with_tools(
            crate::core::chat_templates::QWEN3_CHATML,
            &msgs,
            None,
            true,
            None,
        )
        .unwrap();
        assert!(!without.contains("Turn-1 reasoning"), "without={without}");
    }

    #[test]
    fn iter229_ac4b_absent_kwargs_render_byte_identical_golden() {
        // Golden pin from spike scenario C1 (pre-change render for a
        // no-tools/no-reasoning transcript is identical by construction:
        // the context gains no new keys and the env additions are only
        // consulted by templates that use them).
        let msgs = vec![
            i229_msg("system", "You are an agent."),
            i229_msg("user", "Fix the bug in foo.rs"),
            i229_msg("assistant", "Done, fixed the panic."),
            i229_msg("user", "Now add a test for it"),
        ];
        let out = render_chat_prompt_with_tools(
            crate::core::chat_templates::QWEN3_CHATML,
            &msgs,
            None,
            true,
            None,
        )
        .unwrap();
        let golden = "<|im_start|>system\nYou are an agent.<|im_end|>\n\
                      <|im_start|>user\nFix the bug in foo.rs<|im_end|>\n\
                      <|im_start|>assistant\nDone, fixed the panic.<|im_end|>\n\
                      <|im_start|>user\nNow add a test for it<|im_end|>\n\
                      <|im_start|>assistant\n<think>\n";
        assert_eq!(out, golden);
        // And kwargs=Some(empty) must equal kwargs=None.
        let empty = serde_json::Map::new();
        let out_empty = render_chat_prompt_with_tools(
            crate::core::chat_templates::QWEN3_CHATML,
            &msgs,
            None,
            true,
            Some(&empty),
        )
        .unwrap();
        assert_eq!(out, out_empty);
    }

    #[test]
    fn iter229_ac4c_kwargs_enable_thinking_overrides_resolved_default() {
        let msgs = vec![i229_msg("user", "hi")];
        let mut kwargs = serde_json::Map::new();
        kwargs.insert("enable_thinking".into(), serde_json::Value::Bool(false));
        // Renderer default says thinking ON; kwargs must win → the qwen
        // template emits the pre-closed think suppressor.
        let out = render_chat_prompt_with_tools(
            crate::core::chat_templates::QWEN3_CHATML,
            &msgs,
            None,
            true,
            Some(&kwargs),
        )
        .unwrap();
        assert!(out.ends_with("<think>\n\n</think>\n\n"), "out={out}");
        // Inverse: default OFF, kwargs ON → open think block.
        kwargs.insert("enable_thinking".into(), serde_json::Value::Bool(true));
        let out = render_chat_prompt_with_tools(
            crate::core::chat_templates::QWEN3_CHATML,
            &msgs,
            None,
            false,
            Some(&kwargs),
        )
        .unwrap();
        assert!(out.ends_with("<think>\n"), "out={out}");
        assert!(!out.ends_with("</think>\n\n"), "out={out}");
    }

    #[test]
    fn iter229_ac4d_reserved_kwargs_rejected_naming_the_key() {
        let msgs = vec![i229_msg("user", "hi")];
        for key in [
            "messages",
            "tools",
            "add_generation_prompt",
            "bos_token",
            "eos_token",
            "raise_exception",
        ] {
            let mut kwargs = serde_json::Map::new();
            kwargs.insert(key.into(), serde_json::Value::Bool(true));
            let err = render_chat_prompt_with_tools(
                crate::core::chat_templates::QWEN3_CHATML,
                &msgs,
                None,
                false,
                Some(&kwargs),
            )
            .unwrap_err();
            let text = format!("{err:#}");
            assert!(
                text.contains("reserved chat_template_kwargs key") && text.contains(key),
                "key={key} err={text}"
            );
        }
    }

    #[test]
    fn iter229_ac4e_free_kwargs_pass_through_verbatim() {
        let msgs = vec![i229_msg("user", "hi")];
        let mut kwargs = serde_json::Map::new();
        kwargs.insert(
            "custom_flag".into(),
            serde_json::Value::String("x{y\"z".into()),
        );
        let out =
            render_chat_prompt_with_tools("{{ custom_flag }}", &msgs, None, false, Some(&kwargs))
                .unwrap();
        assert_eq!(out, "x{y\"z");
    }

    #[test]
    fn iter229_ac6_id_to_name_chronological_scoping() {
        let inspect = "{% for m in messages %}{% if m.tool_responses %}\
                       {% for tr in m.tool_responses %}TR:{{ tr.name }};{% endfor %}\
                       {% endif %}{% endfor %}";
        // (a) duplicate id: each tool message binds to the most recent
        // PRIOR definition.
        let mut a1 = i229_msg("assistant", "");
        a1.tool_calls = Some(vec![i229_tool_call("X", "first_fn", "{}")]);
        let mut t1 = i229_msg("tool", "r1");
        t1.tool_call_id = Some("X".into());
        let mut a2 = i229_msg("assistant", "");
        a2.tool_calls = Some(vec![i229_tool_call("X", "second_fn", "{}")]);
        let mut t2 = i229_msg("tool", "r2");
        t2.tool_call_id = Some("X".into());
        let msgs = vec![i229_msg("user", "go"), a1, t1, a2, t2];
        let out = render_chat_prompt_with_tools(inspect, &msgs, None, false, None).unwrap();
        assert_eq!(out, "TR:first_fn;TR:second_fn;");

        // (b) forward reference: tool message BEFORE the defining
        // assistant turn → "unknown".
        let mut t0 = i229_msg("tool", "early");
        t0.tool_call_id = Some("Y".into());
        let mut a3 = i229_msg("assistant", "");
        a3.tool_calls = Some(vec![i229_tool_call("Y", "late_fn", "{}")]);
        let msgs = vec![i229_msg("user", "go"), t0, a3];
        let out = render_chat_prompt_with_tools(inspect, &msgs, None, false, None).unwrap();
        assert_eq!(out, "TR:unknown;");

        // (c) unknown id → "unknown".
        let mut tz = i229_msg("tool", "orphan");
        tz.tool_call_id = Some("Z".into());
        let msgs = vec![i229_msg("user", "go"), tz];
        let out = render_chat_prompt_with_tools(inspect, &msgs, None, false, None).unwrap();
        assert_eq!(out, "TR:unknown;");

        // (d) tool message with NO tool_call_id: bare {role, content} —
        // no tool_responses synthesized; qwen template consumes content.
        let bare = i229_msg("tool", "bare result");
        let msgs = vec![i229_msg("user", "go"), bare];
        let out = render_chat_prompt_with_tools(inspect, &msgs, None, false, None).unwrap();
        assert_eq!(out, "", "no tool_responses must be synthesized");
        let out = render_chat_prompt_with_tools(
            crate::core::chat_templates::QWEN3_CHATML,
            &msgs,
            None,
            false,
            None,
        )
        .unwrap();
        assert!(
            out.contains("<tool_response>\nbare result\n</tool_response>"),
            "out={out}"
        );
    }

    #[test]
    fn iter229_ac5_raise_sites_surface_template_message() {
        // Renderer-level: each qwen raise site produces an Err whose chain
        // carries the template's own message (Strict policy). The
        // handler's `{e:#}` mapping (render_and_tokenize_for_overflow)
        // forwards this chain into the 400 body.
        let tmpl = crate::core::chat_templates::QWEN3_CHATML;
        let cases: Vec<(Vec<ChatMessage>, &str)> = vec![
            // line 43: no messages at all
            (vec![], "No messages provided"),
            // line 79: no user query anywhere
            (vec![i229_msg("system", "s")], "No user query found"),
            // line 85: system not at the beginning
            (
                vec![i229_msg("user", "u"), i229_msg("system", "late")],
                "System message must be at the beginning",
            ),
            // line 144: unexpected role
            (
                vec![i229_msg("user", "u"), i229_msg("narrator", "x")],
                "Unexpected message role",
            ),
        ];
        for (msgs, expect) in cases {
            let err = render_chat_prompt_with_tools(tmpl, &msgs, None, false, None).unwrap_err();
            let text = format!("{err:#}");
            assert!(
                text.contains(expect),
                "expected {expect:?} in err chain: {text}"
            );
        }
    }

    // -----------------------------------------------------------------
    // Engine::shutdown — joins the worker thread (Phase 2a Decision #17)
    //
    // These tests stand up an `Engine` with a stub worker thread that
    // drains the channel and exits on the `Shutdown` sentinel. The real
    // worker (`worker_run`) requires a `LoadedModel` (GGUF + tokenizer);
    // for unit-testing the lifecycle wiring we substitute a no-op worker
    // that exercises the same exit path. The point of the test is to
    // verify that `Engine::shutdown` actually joins the OS thread, that
    // it is idempotent, and that it propagates a panic in the worker.
    // -----------------------------------------------------------------

    fn make_test_engine_with_worker<F>(worker: F) -> Engine
    where
        F: FnOnce(mpsc::Receiver<Request>) + Send + 'static,
    {
        make_test_engine_with_worker_and_arch(LoadedArch::Gemma, worker)
    }

    /// Iter-215 Wedge-2: same as `make_test_engine_with_worker` but
    /// lets the test specify the `LoadedArch`.  Used by Qwen3.5/3.6
    /// 501 tests to build a synthetic engine reporting
    /// `LoadedArch::Qwen35` without a real GGUF on disk.
    fn make_test_engine_with_worker_and_arch<F>(arch: LoadedArch, worker: F) -> Engine
    where
        F: FnOnce(mpsc::Receiver<Request>) + Send + 'static,
    {
        make_test_engine_with_worker_arch_and_budget(arch, 0, 0, worker)
    }

    /// **ADR-040 §3.5 iter-A5b** — synthetic test engine constructor
    /// with explicit per-slot KV budget + cached per-token bytes. Used
    /// by the iter-A5b tests that exercise `Engine::try_admit_budget`
    /// + the handler-side `slot_budget_exceeded` routing.
    ///
    /// Both `per_slot_kv_budget_bytes = 0` and `kv_bytes_per_token = 0`
    /// preserve the pre-A5 byte-equivalence (enforcement disabled);
    /// any non-zero value on BOTH fields exercises the typed
    /// `EngineAdmitError::SlotBudgetExceeded` path.
    fn make_test_engine_with_worker_arch_and_budget<F>(
        arch: LoadedArch,
        per_slot_kv_budget_bytes: u64,
        kv_bytes_per_token_cached: u64,
        worker: F,
    ) -> Engine
    where
        F: FnOnce(mpsc::Receiver<Request>) + Send + 'static,
    {
        let (tx, rx) = mpsc::channel::<Request>(8);
        let handle = std::thread::Builder::new()
            .name("hf2q-engine-test".into())
            .spawn(move || worker(rx))
            .expect("spawn test worker");

        Engine {
            inner: Arc::new(EngineInner {
                tx,
                worker_handle: Mutex::new(Some(handle)),
                info: synthetic_load_info("test-model"),
                arch,
                model_id: "test-model".into(),
                context_length: None,
                quant_type: None,
                hidden_size: 0,
                vocab_size: 0,
                eos_token_ids: vec![],
                tokenizer: Arc::new(Tokenizer::new(tokenizers::models::bpe::BPE::default())),
                chat_template: Arc::new(String::new()),
                registration: None,
                token_bytes: std::sync::OnceLock::new(),
                kv_spill_descriptor: None,
                tq_packed_descriptor: None,
                mode: EngineMode::SerialFifo,
                // ADR-040 C2b scaffold for test-only engines.
                max_slots: 1,
                per_slot_kv_budget_bytes,
                kv_bytes_per_token_cached,
                scheduler_stats_snapshot: Arc::new(Mutex::new(SchedulerStats {
                    policy: SchedulerPolicy::FifoSerial,
                    in_flight_slots: 0,
                    queue_capacity: 8,
                    admitted_total: 0,
                    rejected_429_total: 0,
                    completed_total: 0,
                })),
            }),
        }
    }

    #[test]
    fn engine_info_returns_populated_load_info() {
        let engine = make_test_engine_with_worker(drain_until_shutdown);
        let info = engine.info();
        assert_eq!(info.model_id, "test-model");
        assert_eq!(info.arch_family, ArchFamily::Gemma4);
        assert_eq!(info.backend, "mlx-native");
        assert_eq!(info.tokenizer_source, TokenizerSource::GgufEmbedded);
        assert_eq!(info.chat_template_source, ChatTemplateSource::None);
        assert_eq!(
            info.provenance,
            crate::core::provenance::Provenance::External
        );
        assert_eq!(info.load_wall_clock, Duration::ZERO);
        let rt = tokio::runtime::Builder::new_current_thread()
            .enable_all()
            .build()
            .expect("test runtime");
        rt.block_on(engine.shutdown()).expect("shutdown");
    }

    /// Stub worker: drain until `Shutdown`, then exit cleanly.
    fn drain_until_shutdown(mut rx: mpsc::Receiver<Request>) {
        while let Some(req) = rx.blocking_recv() {
            if matches!(req, Request::Shutdown) {
                break;
            }
            // Other request kinds are not exercised by these tests; in the
            // production worker they have replies, but here we just drop
            // them — the senders never await a reply.
        }
    }

    // ─────────────────────────────────────────────────────────────────────
    // cfa-iter-A5b CRITICAL #1 — Engine::try_admit_budget pre-stream check
    // tests. These prove the typed-error seam actually fires under a
    // synthetic budget; the production hot path (worker_run) flows the
    // same numbers in via `Engine::spawn`.
    // ─────────────────────────────────────────────────────────────────────

    #[test]
    fn a5b_try_admit_budget_returns_ok_under_zero_budget() {
        // Both fields 0 ⇒ enforcement disabled ⇒ Ok regardless of
        // prompt + max_tokens.
        let engine = make_test_engine_with_worker_arch_and_budget(
            LoadedArch::Gemma,
            0,
            0,
            drain_until_shutdown,
        );
        assert!(engine.try_admit_budget(0, 0).is_ok());
        assert!(
            engine.try_admit_budget(u32::MAX, u32::MAX).is_ok(),
            "u64::MAX-saturating needed_bytes still admits under zero budget"
        );
    }

    #[test]
    fn a5b_try_admit_budget_returns_ok_when_only_budget_set() {
        // Budget set but kv_bytes_per_token = 0 (synthetic loader) ⇒
        // Ok (cannot compute cost; treat as do-not-enforce per
        // scheduler.rs `kv_bytes_needed: 0` opt-out).
        let engine = make_test_engine_with_worker_arch_and_budget(
            LoadedArch::Gemma,
            1024 * 1024,
            0,
            drain_until_shutdown,
        );
        assert!(engine.try_admit_budget(1000, 1000).is_ok());
    }

    #[test]
    fn a5b_try_admit_budget_returns_ok_when_only_per_token_set() {
        // Per-token set but budget = 0 ⇒ Ok (operator didn't pass
        // --kv-cache-budget-bytes; preserves pre-A5 byte-equivalence).
        let engine = make_test_engine_with_worker_arch_and_budget(
            LoadedArch::Gemma,
            0,
            4096,
            drain_until_shutdown,
        );
        assert!(engine.try_admit_budget(1000, 1000).is_ok());
    }

    #[test]
    fn a5b_try_admit_budget_returns_ok_under_budget() {
        // 1 MiB budget, 256 bytes/token, 100 + 100 = 200 tokens ⇒
        // 200 × 256 = 51_200 bytes ≤ 1_048_576 ⇒ Ok.
        let engine = make_test_engine_with_worker_arch_and_budget(
            LoadedArch::Gemma,
            1024 * 1024,
            256,
            drain_until_shutdown,
        );
        assert!(engine.try_admit_budget(100, 100).is_ok());
    }

    #[test]
    fn a5b_try_admit_budget_returns_slot_budget_exceeded_when_over() {
        // 1 MiB budget, 1024 bytes/token, 1000 + 1000 = 2000 tokens ⇒
        // 2000 × 1024 = 2_048_000 > 1_048_576 ⇒ SlotBudgetExceeded.
        let engine = make_test_engine_with_worker_arch_and_budget(
            LoadedArch::Gemma,
            1024 * 1024,
            1024,
            drain_until_shutdown,
        );
        match engine.try_admit_budget(1000, 1000) {
            Err(EngineAdmitError::SlotBudgetExceeded {
                needed_bytes,
                budget_bytes,
            }) => {
                assert_eq!(
                    needed_bytes,
                    2000 * 1024,
                    "needed_bytes = (prompt + max) × per_token"
                );
                assert_eq!(
                    budget_bytes,
                    1024 * 1024,
                    "budget_bytes echoes the configured per-slot budget"
                );
            }
            Ok(()) => panic!("over-budget admit MUST surface SlotBudgetExceeded, got Ok"),
        }
    }

    #[test]
    fn a5b_try_admit_budget_at_budget_exactly_returns_ok() {
        // Boundary: needed == budget is Ok (strict `>` in the check).
        let engine = make_test_engine_with_worker_arch_and_budget(
            LoadedArch::Gemma,
            1024 * 1024,
            1024,
            drain_until_shutdown,
        );
        // 1024 × 1024 = 1 MiB exactly.
        assert!(
            engine.try_admit_budget(512, 512).is_ok(),
            "at-budget admit (needed == budget) must succeed"
        );
    }

    #[test]
    fn a5b_per_slot_kv_budget_bytes_accessor_echoes_stored_value() {
        // The accessor is the load-bearing surface for Prometheus
        // exposition + future per-slot operator views; pin that it
        // simply echoes EngineInner.per_slot_kv_budget_bytes.
        let engine = make_test_engine_with_worker_arch_and_budget(
            LoadedArch::Gemma,
            4 * 1024 * 1024,
            256,
            drain_until_shutdown,
        );
        assert_eq!(engine.per_slot_kv_budget_bytes(), 4 * 1024 * 1024);
        let engine0 = make_test_engine_with_worker_arch_and_budget(
            LoadedArch::Gemma,
            0,
            0,
            drain_until_shutdown,
        );
        assert_eq!(
            engine0.per_slot_kv_budget_bytes(),
            0,
            "0 means enforcement disabled"
        );
    }

    #[test]
    fn a5b_engine_admit_error_display_names_needed_and_budget() {
        // The Display impl is what the operator sees in tracing logs +
        // anyhow chains; pin that it names both fields verbatim AND
        // cites ADR-040 §3.5.
        let err = EngineAdmitError::SlotBudgetExceeded {
            needed_bytes: 12_345_678,
            budget_bytes: 4_096_000,
        };
        let display = format!("{err}");
        assert!(
            display.contains("12345678"),
            "Display names needed_bytes verbatim: {display}"
        );
        assert!(
            display.contains("4096000"),
            "Display names budget_bytes verbatim: {display}"
        );
        assert!(
            display.contains("ADR-040"),
            "Display cites ADR-040: {display}"
        );
        assert!(
            display.contains("max_tokens") || display.contains("prompt"),
            "Display names the actionable remediation: {display}"
        );
    }

    // ─────────────────────────────────────────────────────────────────────
    // **DOWNGRADED to seam-only per iter-A5d (cfa-iter-A5c BLOCK closure)**
    //
    // Codex /cfa BLOCK verdicts on iter-A5b AND iter-A5c flagged these
    // tests as "seam-level, NOT handler-level". They do NOT call
    // `chat_completions` or `chat_completions_stream`; they synthesise the
    // `Response` from `ApiError::slot_budget_exceeded(...).into_response()`
    // after calling `engine.try_admit_budget(...)` directly. That proved
    // the ApiError wire shape — but NOT handler routing, request
    // extraction, PreparedChatContext wiring, or actual pre-SSE handler
    // behaviour.
    //
    // Iter-A5d closes Critical #2 with REAL handler-level tests at
    // `src/serve/api/handlers.rs::a5d_handler_429_tests`:
    //   - `a5d_chat_completions_stream_handler_returns_429_application_json_not_sse_when_kv_budget_exceeded`
    //   - `a5d_chat_completions_non_streaming_handler_returns_429_when_worker_signals_slot_budget_exceeded`
    //
    // These invoke the EXACT production handler functions
    // (`chat_completions_stream` and the iter-A5d-extracted
    // `chat_completions_with_prepared`) with a synthetic over-budget
    // `Engine` + a real `AppState` + a real `PreparedChatContext`.
    // That is the load-bearing closure for Critical #2.
    //
    // The tests below are RETAINED as **supplemental** seam-only
    // proofs of the ApiError wire shape (independent of handler
    // routing) + the structural source-order pin. They were renamed
    // `a5d_seam_only_*` per iter-A5d so the test-name prefix surfaces
    // their actual scope. Operators reading test names see the truth:
    //   - `a5d_seam_only_*` = ApiError + try_admit_budget seam shape
    //   - `a5d_*_handler_returns_429_*` = production handler call
    // ─────────────────────────────────────────────────────────────────────

    /// **SEAM-ONLY (supplemental to `a5d_chat_completions_non_streaming_handler_returns_429_*`)**
    /// — direct `engine.try_admit_budget(...)` + `ApiError::slot_budget_exceeded`
    /// wire-shape proof. Does NOT invoke `chat_completions` or any handler.
    ///
    /// Retained because it isolates the
    /// `EngineAdmitError::SlotBudgetExceeded → ApiError::slot_budget_exceeded → 429+JSON`
    /// chain without the handler routing in between, so a regression
    /// in JUST the seam (without breaking the handler test) would
    /// still surface here with a precise failure message.
    ///
    /// Falsifier path (any one ⇒ seam contract broken):
    /// 1. `try_admit_budget` returns Ok for an over-budget request.
    /// 2. The error variant doesn't carry both needed + budget.
    /// 3. `ApiError::slot_budget_exceeded` produces a non-429 status.
    /// 4. The response lacks `Retry-After: 1`.
    /// 5. The body JSON does not contain `"code":"slot_budget_exceeded"`.
    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
    async fn a5d_seam_only_try_admit_budget_to_api_error_429_wire_shape() {
        use super::super::schema::ApiError;
        use axum::body::to_bytes;
        use axum::http::header;
        use axum::response::IntoResponse;

        // Synthetic Engine mirroring the production `Engine::spawn`
        // shape: per_slot_kv_budget_bytes = 1 MiB; kv_bytes_per_token =
        // 1 KiB. A request asking for (prompt=1000 + max_tokens=1000)
        // tokens needs 2000 × 1024 = 2 MiB > 1 MiB budget.
        let engine = make_test_engine_with_worker_arch_and_budget(
            LoadedArch::Gemma,
            1024 * 1024,
            1024,
            drain_until_shutdown,
        );

        // EXACTLY mirrors the non-streaming handler path at
        // handlers.rs:447-449 — except that path goes through worker_run
        // and string-matches; this path uses the `try_admit_budget` seam.
        // The streaming path (CRITICAL #2's load-bearing case) uses
        // `try_admit_budget` directly.
        let response = match engine.try_admit_budget(1000, 1000) {
            Err(EngineAdmitError::SlotBudgetExceeded {
                needed_bytes,
                budget_bytes,
            }) => ApiError::slot_budget_exceeded(needed_bytes, budget_bytes).into_response(),
            Ok(()) => panic!(
                "Synthetic over-budget admit MUST surface SlotBudgetExceeded — \
                 indicates `try_admit_budget` is broken"
            ),
        };

        // (1/5) HTTP 429.
        assert_eq!(
            response.status(),
            axum::http::StatusCode::TOO_MANY_REQUESTS,
            "non-streaming over-budget MUST return 429"
        );
        // (2/5) Retry-After: 1.
        let retry_after = response.headers().get(header::RETRY_AFTER);
        assert_eq!(
            retry_after.and_then(|v| v.to_str().ok()),
            Some("1"),
            "non-streaming over-budget MUST set Retry-After: 1"
        );
        // (3/5) JSON body shape — Content-Type + code field.
        let ct = response
            .headers()
            .get(header::CONTENT_TYPE)
            .and_then(|v| v.to_str().ok())
            .unwrap_or("");
        assert!(
            ct.contains("application/json"),
            "body Content-Type MUST be application/json; got {ct:?}"
        );
        let body_bytes = to_bytes(response.into_body(), 1 << 20)
            .await
            .expect("collect body bytes");
        let body_str = String::from_utf8_lossy(&body_bytes).into_owned();
        // (4/5) Body contains `slot_budget_exceeded` code.
        assert!(
            body_str.contains("slot_budget_exceeded"),
            "body MUST contain `slot_budget_exceeded` code; got: {body_str}"
        );
        // (5/5) Body contains the actual byte numbers (operator-facing
        // remediation diagnostic — parse_slot_budget_exceeded contract
        // depends on these being verbatim in the message).
        assert!(
            body_str.contains("2048000"),
            "body MUST embed needed_bytes=2048000 verbatim; got: {body_str}"
        );
        assert!(
            body_str.contains("1048576"),
            "body MUST embed budget_bytes=1048576 verbatim; got: {body_str}"
        );

        engine.shutdown().await.expect("shutdown");
    }

    /// **CRITICAL #2 GOLDEN** — streaming wire-shape: a streaming chat
    /// **SEAM-ONLY (supplemental to `a5d_chat_completions_stream_handler_returns_429_application_json_not_sse_*`)**
    /// — direct `engine.try_admit_budget(...)` + `ApiError::slot_budget_exceeded`
    /// wire-shape proof for the streaming-path response. Does NOT
    /// invoke `chat_completions_stream` or any handler.
    ///
    /// Retained because it isolates the `application/json` vs
    /// `text/event-stream` Content-Type discriminator at the seam
    /// (independent of handler routing). The true load-bearing
    /// handler-level test lives at
    /// `handlers.rs::a5d_handler_429_tests::a5d_chat_completions_stream_handler_returns_429_application_json_not_sse_when_kv_budget_exceeded`.
    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
    async fn a5d_seam_only_streaming_response_is_json_not_sse_when_over_budget() {
        use super::super::schema::ApiError;
        use axum::http::header;
        use axum::response::IntoResponse;

        let engine = make_test_engine_with_worker_arch_and_budget(
            LoadedArch::Gemma,
            1024 * 1024,
            1024,
            drain_until_shutdown,
        );

        // EXACTLY mirrors the streaming handler path at
        // handlers.rs:1748-1766 — `try_admit_budget` returns BEFORE the
        // handler reaches the SSE-building `generate_stream_with_deepstack`
        // call at handlers.rs:1767. The response produced via
        // `ApiError::slot_budget_exceeded(...)` is `application/json`,
        // not `text/event-stream`.
        let response = match engine.try_admit_budget(1000, 1000) {
            Err(EngineAdmitError::SlotBudgetExceeded {
                needed_bytes,
                budget_bytes,
            }) => ApiError::slot_budget_exceeded(needed_bytes, budget_bytes).into_response(),
            Ok(()) => panic!(
                "Synthetic over-budget admit MUST surface SlotBudgetExceeded \
                 — streaming pre-admit seam is broken"
            ),
        };

        // Load-bearing assertion #1: 429 (not 200 + mid-stream error).
        assert_eq!(
            response.status(),
            axum::http::StatusCode::TOO_MANY_REQUESTS,
            "streaming over-budget MUST short-circuit to 429 PRE-SSE; \
             a 200 here would indicate the handler proceeded to SSE body \
             construction and surfaced the error mid-stream (the iter-A5 \
             defect codex flagged)"
        );
        // Load-bearing assertion #2: Content-Type is JSON, NOT
        // text/event-stream. If the SSE body had been constructed, the
        // response would carry `text/event-stream`.
        let ct = response
            .headers()
            .get(header::CONTENT_TYPE)
            .and_then(|v| v.to_str().ok())
            .unwrap_or("");
        assert!(
            ct.contains("application/json"),
            "streaming pre-admit 429 MUST be a JSON body (NOT \
             text/event-stream); got Content-Type: {ct:?} — this assertion \
             is the wire-level proof that the response is NOT an SSE \
             body"
        );
        assert!(
            !ct.contains("text/event-stream"),
            "streaming pre-admit 429 MUST NOT carry text/event-stream; \
             got: {ct:?}"
        );
        // Retry-After: 1 — matches queue_full convention.
        let retry_after = response.headers().get(header::RETRY_AFTER);
        assert_eq!(
            retry_after.and_then(|v| v.to_str().ok()),
            Some("1"),
            "streaming over-budget MUST set Retry-After: 1 (parallel to \
             queue_full convention)"
        );

        engine.shutdown().await.expect("shutdown");
    }

    /// **SOURCE-ORDER GREP (supplemental to handler-level tests)** —
    /// structural pin: the streaming handler at
    /// `handlers.rs::chat_completions_stream` calls
    /// `engine.try_admit_budget` BEFORE it calls
    /// `generate_stream_with_deepstack`.
    ///
    /// This test is purely supplemental: a refactor that swaps the order
    /// would also break the real handler-level test
    /// (`a5d_chat_completions_stream_handler_returns_429_application_json_not_sse_when_kv_budget_exceeded`)
    /// because Content-Type would become `text/event-stream`. The source
    /// grep here gives an additional, faster-to-diagnose failure mode
    /// (lints the source order BEFORE the handler test executes the
    /// actual code), so it is retained per iter-A5d.
    #[test]
    fn a5d_seam_only_streaming_handler_admit_check_precedes_stream_call_source_grep() {
        let handlers_src = include_str!("handlers.rs");
        // Locate `async fn chat_completions_stream` — the streaming
        // handler — and slice from that fn header to the next top-level
        // function. The two production landmarks within this slice are:
        //   - `engine.try_admit_budget(` (the pre-admit seam call).
        //   - `engine.generate_stream_with_deepstack(` (the SSE-body
        //     construction call).
        let fn_start = handlers_src
            .find("async fn chat_completions_stream(")
            .expect("source must contain `async fn chat_completions_stream(`");
        // Slice to a generous upper bound — the next `pub async fn` or
        // `async fn` 200_000 chars on (the stream fn body is well under
        // 200KB even with macro expansion).
        let upper = (fn_start + 200_000).min(handlers_src.len());
        let slice = &handlers_src[fn_start..upper];

        let admit_pos = slice.find("engine.try_admit_budget(").expect(
            "`engine.try_admit_budget(` call MUST exist in \
                 chat_completions_stream — pre-stream admit seam (codex \
                 CRITICAL #2 fix at iter-A5b)",
        );
        let stream_pos = slice
            .find("engine\n        .generate_stream_with_deepstack")
            .or_else(|| slice.find(".generate_stream_with_deepstack("))
            .expect(
                "`.generate_stream_with_deepstack(` call MUST exist in \
                 chat_completions_stream",
            );

        assert!(
            admit_pos < stream_pos,
            "CRITICAL #2 wire ordering: `engine.try_admit_budget` (byte \
             offset {admit_pos} within chat_completions_stream body) MUST \
             precede `.generate_stream_with_deepstack` (byte offset \
             {stream_pos}); a refactor that reorders these would regress \
             the pre-stream 429 contract pinned by the iter-A5d \
             handler-level test \
             `a5d_chat_completions_stream_handler_returns_429_application_json_not_sse_when_kv_budget_exceeded` \
             in `src/serve/api/handlers.rs`."
        );
    }

    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
    async fn shutdown_joins_worker_thread() {
        let engine = make_test_engine_with_worker(drain_until_shutdown);
        // Worker should be live before shutdown.
        {
            let guard = engine.inner.worker_handle.lock().unwrap();
            assert!(guard.is_some(), "worker handle present pre-shutdown");
        }
        engine.shutdown().await.expect("clean shutdown");
        // Handle must have been taken (and joined) by shutdown.
        let guard = engine.inner.worker_handle.lock().unwrap();
        assert!(guard.is_none(), "worker handle taken post-shutdown");
    }

    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
    async fn shutdown_is_idempotent() {
        let engine = make_test_engine_with_worker(drain_until_shutdown);
        engine.shutdown().await.expect("first shutdown");
        // A second call must not panic and must not deadlock; the
        // worker_handle slot is empty so the join step is skipped, and
        // the Sender is closed so `tx.send` errors silently.
        engine.shutdown().await.expect("second shutdown is no-op");
    }

    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
    async fn shutdown_propagates_worker_panic() {
        // Worker panics on its first message. shutdown() sends Shutdown,
        // which the worker receives → panics → join returns Err → our
        // shutdown() returns Err with the panic context.
        let engine = make_test_engine_with_worker(|mut rx| {
            let _ = rx.blocking_recv();
            panic!("test panic in worker");
        });
        let res = engine.shutdown().await;
        let err = res.expect_err("expected join failure");
        let msg = format!("{}", err);
        assert!(
            msg.contains("panicked"),
            "shutdown error should name 'panicked', got: {msg}"
        );
    }

    // -----------------------------------------------------------------------
    // Wave-2.5 B5 — PromptCache key expansion (HIGH-7 silent-correctness fix)
    // -----------------------------------------------------------------------

    /// Helper: build a stored PromptCache that looks like a previous greedy
    /// request completed with `result_text`.
    fn make_cached(tokens: &[u32], params: &SamplingParams, result_text: &str) -> PromptCache {
        let mut cache = PromptCache::new();
        let result = GenerationResult {
            text: result_text.to_string(),
            reasoning_text: None,
            prompt_tokens: tokens.len(),
            completion_tokens: 5,
            reasoning_tokens: None,
            finish_reason: "stop",
            prefill_duration: Duration::ZERO,
            decode_duration: Duration::ZERO,
            cached_tokens: 0,
            logprobs: None,
        };
        cache.store(tokens, params, &result);
        cache
    }

    #[test]
    fn prompt_cache_miss_on_different_max_tokens() {
        let tokens: Vec<u32> = vec![1, 2, 3];
        let mut base = SamplingParams::default();
        base.max_tokens = 100;
        let cache = make_cached(&tokens, &base, "hello");

        let mut req = SamplingParams::default();
        req.max_tokens = 200; // different max_tokens — must MISS
        assert!(
            cache.lookup(&tokens, &req).is_none(),
            "same prompt + different max_tokens must not hit cache"
        );
    }

    #[test]
    fn prompt_cache_miss_on_different_stop_strings() {
        let tokens: Vec<u32> = vec![1, 2, 3];
        let mut base = SamplingParams::default();
        base.stop_strings = vec!["STOP".to_string()];
        let cache = make_cached(&tokens, &base, "hello");

        let mut req = SamplingParams::default();
        req.stop_strings = vec!["END".to_string()]; // different stops — must MISS
        assert!(
            cache.lookup(&tokens, &req).is_none(),
            "same prompt + different stop_strings must not hit cache"
        );
    }

    #[test]
    fn prompt_cache_miss_on_different_logit_bias() {
        let tokens: Vec<u32> = vec![1, 2, 3];
        let mut base = SamplingParams::default();
        base.logit_bias.insert(42, 5.0);
        let cache = make_cached(&tokens, &base, "hello");

        let mut req = SamplingParams::default();
        req.logit_bias.insert(42, 10.0); // different bias value — must MISS
        assert!(
            cache.lookup(&tokens, &req).is_none(),
            "same prompt + different logit_bias must not hit cache"
        );
    }

    #[test]
    fn prompt_cache_miss_on_different_response_format_grammar() {
        use super::super::grammar::parser::{Grammar, GretElement, GretType};
        use std::collections::HashMap;

        let tokens: Vec<u32> = vec![1, 2, 3];

        // Build a trivial grammar A: single rule with one Char element + End.
        let grammar_a = Grammar {
            rules: vec![vec![
                GretElement::new(GretType::Char, b'a' as u32),
                GretElement::new(GretType::End, 0),
            ]],
            symbol_ids: {
                let mut m = HashMap::new();
                m.insert("root".to_string(), 0u32);
                m
            },
        };
        // Grammar B differs in the Char value.
        let grammar_b = Grammar {
            rules: vec![vec![
                GretElement::new(GretType::Char, b'b' as u32),
                GretElement::new(GretType::End, 0),
            ]],
            symbol_ids: {
                let mut m = HashMap::new();
                m.insert("root".to_string(), 0u32);
                m
            },
        };

        let mut base = SamplingParams::default();
        base.grammar = Some(grammar_a);
        let cache = make_cached(&tokens, &base, r#"{"ok":true}"#);

        let mut req = SamplingParams::default();
        req.grammar = Some(grammar_b); // different grammar — must MISS
        assert!(
            cache.lookup(&tokens, &req).is_none(),
            "same prompt + different grammar must not hit cache"
        );
    }

    /// Wave-2.5 B5 — tool_choice key sensitivity.
    ///
    /// `tool_choice` is compiled to a `Grammar` (via `compile_tool_grammar`)
    /// before being stored in `SamplingParams.grammar`. Two requests with the
    /// same prompt but different tool grammars (different tool_choice values)
    /// must produce a cache MISS. This test exercises the grammar arm of the
    /// PromptCacheKey directly — the same code path that `tool_choice=function`
    /// exercises at the end of `prepare_chat_completion_common`.
    #[test]
    fn prompt_cache_miss_on_different_tool_choice_grammar() {
        use super::super::grammar::parser::{Grammar, GretElement, GretType};
        use std::collections::HashMap;

        let tokens: Vec<u32> = vec![1, 2, 3];

        // Simulate grammar compiled for tool_choice=function{name:"tool_a"}.
        let grammar_tool_a = Grammar {
            rules: vec![vec![
                GretElement::new(GretType::Char, b'a' as u32),
                GretElement::new(GretType::End, 0),
            ]],
            symbol_ids: {
                let mut m = HashMap::new();
                m.insert("root".to_string(), 0u32);
                m
            },
        };
        // Simulate grammar compiled for tool_choice=function{name:"tool_b"}.
        let grammar_tool_b = Grammar {
            rules: vec![vec![
                GretElement::new(GretType::Char, b'b' as u32),
                GretElement::new(GretType::End, 0),
            ]],
            symbol_ids: {
                let mut m = HashMap::new();
                m.insert("root".to_string(), 0u32);
                m
            },
        };

        // Cache a response generated under tool_a grammar.
        let mut base = SamplingParams::default();
        base.grammar = Some(grammar_tool_a);
        let cache = make_cached(&tokens, &base, r#"{"name":"tool_a"}"#);

        // A subsequent request with tool_b grammar must MISS — it would
        // produce different output under a different constraint.
        let mut req = SamplingParams::default();
        req.grammar = Some(grammar_tool_b);
        assert!(
            cache.lookup(&tokens, &req).is_none(),
            "same prompt + different tool_choice grammar must not hit cache \
             (would silently replay the wrong tool call)"
        );

        // A request with NO grammar (tool_choice absent / unconstrained) must
        // also MISS — unconstrained decode differs from constrained decode.
        let req_no_grammar = SamplingParams::default();
        assert!(
            cache.lookup(&tokens, &req_no_grammar).is_none(),
            "same prompt + no grammar vs. tool grammar must not hit cache"
        );
    }

    #[test]
    fn prompt_cache_hit_requires_all_params_equal() {
        let tokens: Vec<u32> = vec![10, 20, 30];
        let mut params = SamplingParams::default();
        params.max_tokens = 64;
        params.stop_strings = vec!["END".to_string()];
        params.logit_bias.insert(99, -1.0);
        let cache = make_cached(&tokens, &params, "cached text");

        // Same params → must HIT
        let mut same = SamplingParams::default();
        same.max_tokens = 64;
        same.stop_strings = vec!["END".to_string()];
        same.logit_bias.insert(99, -1.0);
        let hit = cache.lookup(&tokens, &same);
        assert!(
            hit.is_some(),
            "identical prompt + identical params must hit cache"
        );
        assert_eq!(hit.unwrap().text, "cached text");
    }

    // -----------------------------------------------------------------------
    // Wave-2.6 W-ε — honest B5 closure: tests for newly-keyed params
    // -----------------------------------------------------------------------

    #[test]
    fn prompt_cache_miss_on_different_grammar_kind() {
        let tokens: Vec<u32> = vec![1, 2, 3];
        use super::super::grammar::parser::{Grammar, GretElement, GretType};
        use std::collections::HashMap;

        // Build a trivial grammar used by both base and req.
        let grammar = Grammar {
            rules: vec![vec![
                GretElement::new(GretType::Char, b'x' as u32),
                GretElement::new(GretType::End, 0),
            ]],
            symbol_ids: {
                let mut m = HashMap::new();
                m.insert("root".to_string(), 0u32);
                m
            },
        };

        let mut base = SamplingParams::default();
        base.grammar = Some(grammar.clone());
        base.grammar_kind = GrammarKind::ResponseFormat;
        let cache = make_cached(&tokens, &base, "x");

        // Same grammar, but ToolCallBodyAuto kind — enforcement timing differs → MISS.
        let mut req = SamplingParams::default();
        req.grammar = Some(grammar);
        req.grammar_kind = GrammarKind::ToolCallBodyAuto;
        assert!(
            cache.lookup(&tokens, &req).is_none(),
            "same grammar + different grammar_kind must not hit cache \
             (ResponseFormat enforces unconditionally; ToolCallBodyAuto is trigger-gated)"
        );
    }

    #[test]
    fn prompt_cache_miss_on_different_frequency_penalty() {
        let tokens: Vec<u32> = vec![1, 2, 3];
        let mut base = SamplingParams::default();
        base.frequency_penalty = 0.0;
        let cache = make_cached(&tokens, &base, "hello");

        let mut req = SamplingParams::default();
        req.frequency_penalty = 0.5; // non-default — must MISS
        assert!(
            cache.lookup(&tokens, &req).is_none(),
            "same prompt + different frequency_penalty must not hit cache"
        );
    }

    #[test]
    fn prompt_cache_miss_on_different_presence_penalty() {
        let tokens: Vec<u32> = vec![1, 2, 3];
        let mut base = SamplingParams::default();
        base.presence_penalty = 0.0;
        let cache = make_cached(&tokens, &base, "hello");

        let mut req = SamplingParams::default();
        req.presence_penalty = 0.3; // non-default — must MISS
        assert!(
            cache.lookup(&tokens, &req).is_none(),
            "same prompt + different presence_penalty must not hit cache"
        );
    }

    #[test]
    fn prompt_cache_miss_on_different_min_p() {
        let tokens: Vec<u32> = vec![1, 2, 3];
        let mut base = SamplingParams::default();
        base.min_p = 0.0;
        let cache = make_cached(&tokens, &base, "hello");

        let mut req = SamplingParams::default();
        req.min_p = 0.1; // non-default — must MISS
        assert!(
            cache.lookup(&tokens, &req).is_none(),
            "same prompt + different min_p must not hit cache"
        );
    }

    #[test]
    fn prompt_cache_miss_on_different_tool_call_policy() {
        let tokens: Vec<u32> = vec![1, 2, 3];
        let mut base = SamplingParams::default();
        base.tool_call_policy = ToolCallPolicy::Auto;
        let cache = make_cached(&tokens, &base, r#"{"name":"fn"}"#);

        let mut req = SamplingParams::default();
        req.tool_call_policy = ToolCallPolicy::Constrained; // different policy — must MISS
        assert!(
            cache.lookup(&tokens, &req).is_none(),
            "same prompt + different tool_call_policy must not hit cache \
             (Constrained promotes parse failures; Auto falls back to content)"
        );
    }

    #[test]
    fn prompt_cache_miss_on_different_logprobs() {
        let tokens: Vec<u32> = vec![1, 2, 3];
        let mut base = SamplingParams::default();
        base.logprobs = false;
        let cache = make_cached(&tokens, &base, "hello");

        let mut req = SamplingParams::default();
        req.logprobs = true; // logprob data requested — different response shape → MISS
        assert!(
            cache.lookup(&tokens, &req).is_none(),
            "same prompt + logprobs=true vs false must not hit cache \
             (response shape differs: logprob entries present vs absent)"
        );
    }

    #[test]
    fn prompt_cache_miss_on_different_top_logprobs() {
        let tokens: Vec<u32> = vec![1, 2, 3];
        let mut base = SamplingParams::default();
        base.logprobs = true;
        base.top_logprobs = 2;
        let cache = make_cached(&tokens, &base, "hello");

        let mut req = SamplingParams::default();
        req.logprobs = true;
        req.top_logprobs = 5; // different number of alternatives — must MISS
        assert!(
            cache.lookup(&tokens, &req).is_none(),
            "same prompt + different top_logprobs must not hit cache \
             (response shape differs: number of top alternatives)"
        );
    }

    #[test]
    fn prompt_cache_miss_on_different_parallel_tool_calls() {
        let tokens: Vec<u32> = vec![1, 2, 3];
        let mut base = SamplingParams::default();
        base.parallel_tool_calls = true;
        let cache = make_cached(&tokens, &base, "hello");

        let mut req = SamplingParams::default();
        req.parallel_tool_calls = false; // non-default — must MISS
        assert!(
            cache.lookup(&tokens, &req).is_none(),
            "same prompt + different parallel_tool_calls must not hit cache"
        );
    }

    /// Full-inventory hit test: all generation-affecting params identical
    /// including the wave-2.6 W-ε additions.
    #[test]
    fn prompt_cache_hit_full_inventory_equal() {
        use super::super::grammar::parser::{Grammar, GretElement, GretType};
        use std::collections::HashMap;

        let tokens: Vec<u32> = vec![10, 20, 30];

        let grammar = Grammar {
            rules: vec![vec![
                GretElement::new(GretType::Char, b'z' as u32),
                GretElement::new(GretType::End, 0),
            ]],
            symbol_ids: {
                let mut m = HashMap::new();
                m.insert("root".to_string(), 0u32);
                m
            },
        };

        let mut params = SamplingParams::default();
        params.max_tokens = 64;
        params.stop_strings = vec!["END".to_string()];
        params.logit_bias.insert(99, -1.0);
        params.grammar = Some(grammar.clone());
        params.grammar_kind = GrammarKind::ResponseFormat;
        params.frequency_penalty = 0.1;
        params.presence_penalty = 0.2;
        params.min_p = 0.05;
        params.tool_call_policy = ToolCallPolicy::Auto;
        params.logprobs = true;
        params.top_logprobs = 3;
        params.parallel_tool_calls = false;
        let cache = make_cached(&tokens, &params, "full-inventory-hit");

        // Identical params → must HIT.
        let mut same = SamplingParams::default();
        same.max_tokens = 64;
        same.stop_strings = vec!["END".to_string()];
        same.logit_bias.insert(99, -1.0);
        same.grammar = Some(grammar);
        same.grammar_kind = GrammarKind::ResponseFormat;
        same.frequency_penalty = 0.1;
        same.presence_penalty = 0.2;
        same.min_p = 0.05;
        same.tool_call_policy = ToolCallPolicy::Auto;
        same.logprobs = true;
        same.top_logprobs = 3;
        same.parallel_tool_calls = false;

        let hit = cache.lookup(&tokens, &same);
        assert!(
            hit.is_some(),
            "identical full-inventory params must hit cache"
        );
        assert_eq!(hit.unwrap().text, "full-inventory-hit");
    }

    // ────────────────────────────────────────────────────────────────
    // ADR-005 iter-224 W-A2.1 — PromptCache fragment-replay scaffolding
    // ────────────────────────────────────────────────────────────────
    //
    // These tests pin the type addition (CachedFragment enum +
    // `fragments: Option<Vec<...>>` field on PromptCache + the new
    // `store_with_fragments` method) without exercising the streaming
    // capture path (W-A2.2) or the replay branch (W-A2.3) yet.
    //
    // Worker AA design report: /tmp/cfa-cfa-audit/prompt-cache-fragment-replay-design.md.

    /// Round-trip: `store(...)` (the legacy single-arg API) must persist
    /// `fragments = None`, matching the non-streaming-origin honest-
    /// minimum behaviour (Worker AA design §3b option (a)).
    #[test]
    fn prompt_cache_store_with_none_fragments_round_trip() {
        let tokens: Vec<u32> = vec![1, 2, 3];
        let params = SamplingParams::default();
        let result = GenerationResult {
            text: "hello world".to_string(),
            reasoning_text: None,
            prompt_tokens: tokens.len(),
            completion_tokens: 7,
            reasoning_tokens: None,
            finish_reason: "stop",
            prefill_duration: Duration::ZERO,
            decode_duration: Duration::ZERO,
            cached_tokens: 0,
            logprobs: None,
        };

        let mut cache = PromptCache::new();
        // Pre-store the fragments slot is None.
        assert!(
            cache.fragments.is_none(),
            "fresh PromptCache must initialise fragments=None"
        );
        cache.store(&tokens, &params, &result);

        assert_eq!(cache.tokens, tokens);
        assert_eq!(cache.text, "hello world");
        assert_eq!(cache.completion_tokens, 7);
        assert_eq!(cache.finish_reason, "stop");
        // Critical: legacy single-arg `store` MUST leave fragments=None
        // so the replay path falls through to the splitter-rerun branch
        // (Wave-3.5 HIGH-2 tail_buf drain preserved).
        assert!(
            cache.fragments.is_none(),
            "store() (legacy single-arg API) must default fragments=None; \
             non-streaming origin has no per-token trace and must use \
             the splitter-rerun replay path (Worker AA design §3b)"
        );

        // Lookup hit shape: fragments are NOT surfaced in GenerationResult
        // — they live alongside text on PromptCache and are consumed by
        // `replay_cached_streaming_response` directly.
        let hit = cache.lookup(&tokens, &params).expect("greedy hit");
        assert_eq!(hit.text, "hello world");
        assert_eq!(hit.cached_tokens, tokens.len());
    }

    /// Round-trip: `store_with_fragments(..., Some(frags))` persists the
    /// captured `Vec<CachedFragment>` byte-for-byte, including all three
    /// variants (Content, Reasoning, ToolCallDelta with both first-chunk
    /// and args-chunk shapes).  This is the W-A2.3 replay-branch input.
    #[test]
    fn prompt_cache_store_with_some_fragments_round_trip() {
        let tokens: Vec<u32> = vec![10, 20, 30, 40];
        let params = SamplingParams::default();
        let result = GenerationResult {
            text: "<thought>plan</thought>call:foo{x:<|\"|>1<|\"|>}".to_string(),
            reasoning_text: None,
            prompt_tokens: tokens.len(),
            completion_tokens: 12,
            reasoning_tokens: Some(2),
            finish_reason: "tool_calls",
            prefill_duration: Duration::ZERO,
            decode_duration: Duration::ZERO,
            cached_tokens: 0,
            logprobs: None,
        };

        // Build a fragment vec exercising all three variants and both
        // ToolCallDelta shapes.
        let frags = vec![
            CachedFragment::Reasoning("plan".to_string()),
            CachedFragment::Content("Here is a result: ".to_string()),
            // First-chunk shape: id+call_type+name set, arguments None.
            CachedFragment::ToolCallDelta {
                index: 0,
                id: Some("call_hf2q_0123456789abcdef".to_string()),
                call_type: Some("function".to_string()),
                name: Some("foo".to_string()),
                arguments: None,
            },
            // Args-chunk shape: only `arguments` populated.
            CachedFragment::ToolCallDelta {
                index: 0,
                id: None,
                call_type: None,
                name: None,
                arguments: Some("{".to_string()),
            },
            CachedFragment::ToolCallDelta {
                index: 0,
                id: None,
                call_type: None,
                name: None,
                arguments: Some("\"x\":1}".to_string()),
            },
        ];

        let mut cache = PromptCache::new();
        cache.store_with_fragments(&tokens, &params, &result, Some(frags.clone()));

        assert_eq!(cache.tokens, tokens);
        assert_eq!(cache.text, result.text);
        assert_eq!(cache.completion_tokens, 12);
        assert_eq!(cache.reasoning_tokens, Some(2));
        assert_eq!(cache.finish_reason, "tool_calls");
        let stored = cache
            .fragments
            .as_ref()
            .expect("Some(fragments) must persist verbatim");
        assert_eq!(
            stored, &frags,
            "stored fragment vec must equal the input vec byte-for-byte; \
             any divergence breaks the W-A2.3 byte-identity contract"
        );

        // Sampling-mode bypass: storing with sampling params (temperature
        // > 0) MUST NOT persist anything — this matches the legacy `store`
        // semantics and prevents a future greedy request from replaying a
        // sampling outcome.
        let mut sampling_params = SamplingParams::default();
        sampling_params.temperature = 0.7;
        let mut cache2 = PromptCache::new();
        cache2.store_with_fragments(&tokens, &sampling_params, &result, Some(frags));
        assert!(
            cache2.tokens.is_empty(),
            "sampling-mode (temperature>0) store_with_fragments must bypass write"
        );
        assert!(
            cache2.fragments.is_none(),
            "sampling-mode bypass leaves fragments at default (None)"
        );
    }

    /// H3 hypothesis pin (Worker AA design §3e): `CachedFragment`
    /// memory footprint is bounded.  Asserts the enum's `size_of` is
    /// reasonable; single-slot cache means total worst-case ~15–20 KB
    /// at 150 fragments — negligible vs. the model itself.
    ///
    /// Rust enum size is determined by the largest variant.  The
    /// `ToolCallDelta` variant carries 4 `Option<String>` (each 24 bytes
    /// with non-null-pointer niche optimisation) + 1 `usize` + tag.
    /// That fixes the union at ~104 bytes today.  The 128-byte budget
    /// includes a small slack for layout-padding shifts.  If a future
    /// edit pushes the enum past this budget, surface the regression
    /// instead of hiding it.
    #[test]
    fn cached_fragment_size_of_is_bounded() {
        let size = std::mem::size_of::<CachedFragment>();
        assert!(
            size <= 128,
            "CachedFragment size_of={} bytes; design budget is ≤128 bytes \
             (Worker AA §3e: single-slot ⇒ ~15–20 KB worst case at 150 frags). \
             A regression past this means the variant layout grew unexpectedly.",
            size
        );
    }

    // ────────────────────────────────────────────────────────────────
    // ADR-005 iter-224 W-A2.2 — streaming-origin capture mechanics
    // ────────────────────────────────────────────────────────────────

    /// Drives `EventSink::with_capture` through a synthetic event stream
    /// and asserts the captured Vec parallels the events forwarded into
    /// the channel — fragment-by-fragment, in order.  This is the
    /// W-A2.2 mechanical regression gate: any future edit that breaks
    /// the "every emit gets captured" invariant fails this test.
    ///
    /// Worker AA design report §3b: streaming origin captures each
    /// emitted `Delta` / `ToolCallDelta` in a sibling `Vec<CachedFragment>`
    /// accumulator.  Live `generate_stream_once` is hard to drive in a
    /// unit test (needs a real `GemmaLoadedModel`); the synthetic test
    /// pins the wrapper's mirror logic at the channel boundary, which
    /// is the same boundary the live decode emits through.
    #[test]
    fn streaming_origin_capture_vec_parallels_emitted_events() {
        use super::super::sse::{DeltaKind, GenerationEvent};

        let (tx, mut rx) = tokio::sync::mpsc::channel::<GenerationEvent>(64);
        let capture: std::cell::RefCell<Vec<CachedFragment>> = std::cell::RefCell::new(Vec::new());
        let sink = EventSink::with_capture(&tx, &capture);

        // Drive a representative sequence: Reasoning + Content + ToolCall
        // first-chunk + ToolCall args + Content postscript + Done + Error.
        // Capture MUST mirror Reasoning / Content / ToolCallDelta.
        // Capture MUST NOT mirror Done / Error / Logprobs.
        sink.blocking_send(GenerationEvent::Delta {
            kind: DeltaKind::Reasoning,
            text: "let me think".to_string(),
        })
        .expect("send 0");
        sink.blocking_send(GenerationEvent::Delta {
            kind: DeltaKind::Content,
            text: "Sure! ".to_string(),
        })
        .expect("send 1");
        sink.blocking_send(GenerationEvent::ToolCallDelta {
            index: 0,
            id: Some("call_hf2q_aaaa".to_string()),
            call_type: Some("function".to_string()),
            name: Some("get_weather".to_string()),
            arguments: None,
        })
        .expect("send 2");
        sink.blocking_send(GenerationEvent::ToolCallDelta {
            index: 0,
            id: None,
            call_type: None,
            name: None,
            arguments: Some("{\"loc\":\"SF\"}".to_string()),
        })
        .expect("send 3");
        sink.blocking_send(GenerationEvent::Delta {
            kind: DeltaKind::Content,
            text: " Done.".to_string(),
        })
        .expect("send 4");
        // Done + Error are control events — captured? No, by design.
        sink.blocking_send(GenerationEvent::Done {
            finish_reason: "tool_calls",
            prompt_tokens: 7,
            completion_tokens: 5,
            stats: super::super::sse::StreamStats::default(),
        })
        .expect("send 5");

        drop(sink);
        drop(tx);

        // Drain channel + collect captured.
        let mut emitted: Vec<GenerationEvent> = Vec::new();
        while let Ok(ev) = rx.try_recv() {
            emitted.push(ev);
        }
        let captured = capture.into_inner();

        // Channel saw all 6 events.
        assert_eq!(
            emitted.len(),
            6,
            "channel must forward all 6 sent events; got {}",
            emitted.len()
        );

        // Capture saw 5 mirror-eligible events (Done is not captured).
        assert_eq!(
            captured.len(),
            5,
            "capture mirrors Delta + ToolCallDelta only — 5 of 6; got {}",
            captured.len()
        );

        // Per-fragment structural check.
        match &captured[0] {
            CachedFragment::Reasoning(t) => assert_eq!(t, "let me think"),
            other => panic!("frag[0]: expected Reasoning; got {other:?}"),
        }
        match &captured[1] {
            CachedFragment::Content(t) => assert_eq!(t, "Sure! "),
            other => panic!("frag[1]: expected Content; got {other:?}"),
        }
        match &captured[2] {
            CachedFragment::ToolCallDelta {
                index,
                id,
                call_type,
                name,
                arguments,
            } => {
                assert_eq!(*index, 0);
                assert_eq!(id.as_deref(), Some("call_hf2q_aaaa"));
                assert_eq!(call_type.as_deref(), Some("function"));
                assert_eq!(name.as_deref(), Some("get_weather"));
                assert!(arguments.is_none());
            }
            other => panic!("frag[2]: expected ToolCallDelta first-chunk; got {other:?}"),
        }
        match &captured[3] {
            CachedFragment::ToolCallDelta {
                index,
                id,
                call_type,
                name,
                arguments,
            } => {
                assert_eq!(*index, 0);
                assert!(id.is_none());
                assert!(call_type.is_none());
                assert!(name.is_none());
                assert_eq!(arguments.as_deref(), Some("{\"loc\":\"SF\"}"));
            }
            other => panic!("frag[3]: expected ToolCallDelta args-chunk; got {other:?}"),
        }
        match &captured[4] {
            CachedFragment::Content(t) => assert_eq!(t, " Done."),
            other => panic!("frag[4]: expected Content; got {other:?}"),
        }
    }

    /// Pin: `EventSink::new(...)` (passive sink) does NOT mirror anything.
    /// This is the property that lets `replay_cached_streaming_response`
    /// and `engine_qwen35::route_content_qwen35` reuse helpers that take
    /// `&EventSink<'_>` without participating in fragment capture.
    #[test]
    fn passive_event_sink_does_not_capture() {
        use super::super::sse::{DeltaKind, GenerationEvent};

        let (tx, mut rx) = tokio::sync::mpsc::channel::<GenerationEvent>(8);
        let sink = EventSink::new(&tx);
        sink.blocking_send(GenerationEvent::Delta {
            kind: DeltaKind::Content,
            text: "hello".to_string(),
        })
        .expect("send");
        drop(sink);
        drop(tx);

        // Channel saw the event.
        assert!(rx.try_recv().is_ok(), "passive sink must still forward");
        // Passive sink has no capture surface — that's the contract.
        // (No assertion needed: there's nothing to inspect because
        // `capture: None`; the contract is structurally enforced.)
    }

    // ────────────────────────────────────────────────────────────────
    // Iter-215 Wedge-2 — LoadedModel enum accessor dispatch
    // ────────────────────────────────────────────────────────────────

    /// LoadedModel accessor methods dispatch correctly to both
    /// variants.  Build synthetic `Gemma` and `Qwen35` instances and
    /// assert each accessor returns the right field for each variant.
    /// Regression guard against a future maintainer adding a field
    /// only to one arm.
    #[test]
    fn loaded_model_enum_accessor_methods_dispatch_correctly() {
        use crate::inference::models::qwen35::{
            default_layer_types, model::Qwen35Model, Qwen35Config, Qwen35MoeConfig, Qwen35Variant,
        };

        // ---- Build a synthetic Qwen35 variant -----------------------
        let cfg = Qwen35Config {
            variant: Qwen35Variant::Moe,
            hidden_size: 64,
            num_hidden_layers: 4,
            num_attention_heads: 4,
            num_key_value_heads: 2,
            head_dim: 16,
            linear_num_key_heads: 4,
            linear_num_value_heads: 8,
            linear_key_head_dim: 16,
            linear_value_head_dim: 16,
            linear_conv_kernel_dim: 4,
            full_attention_interval: 4,
            layer_types: default_layer_types(4, 4),
            partial_rotary_factor: 0.25,
            rope_theta: 1e7,
            rotary_dim: 4,
            mrope_section: [1, 1, 0, 0],
            mrope_interleaved: true,
            rms_norm_eps: 1e-6,
            max_position_embeddings: 1024,
            vocab_size: 256,
            attn_output_gate: true,
            mtp_num_hidden_layers: 0,
            mtp_use_dedicated_embeddings: true,
            intermediate_size: None,
            moe: Some(Qwen35MoeConfig {
                moe_intermediate_size: 16,
                num_experts: 4,
                num_experts_per_tok: 2,
                shared_expert_intermediate_size: 16,
            }),
        };
        let qwen_model = Qwen35Model::empty_from_cfg(cfg);
        let qwen_loaded = super::super::engine_qwen35::Qwen35LoadedModel {
            model: qwen_model,
            tokenizer: Tokenizer::new(tokenizers::models::bpe::BPE::default()),
            chat_template: "qwen-template".to_string(),
            model_id: "qwen-id".to_string(),
            model_path: PathBuf::from("qwen-id.gguf"),
            eos_token_ids: vec![151645],
            hidden_size: 64,
            vocab_size: 256,
            context_length: Some(1024),
            quant_type: Some("Q4_0".to_string()),
            load_duration: Duration::from_millis(7),
            provenance: crate::core::provenance::Provenance::External,
            prompt_cache: super::super::engine_qwen35::HybridPromptCache::new(),
            lcp_registry: crate::serve::kv_persist::lcp_registry::LcpRegistry::new(1),
            kv_metrics_sink: None,
            disk_persistor: None,
            lcp_hydrated_for_cfg: std::collections::HashSet::new(),
            tq_kv_active: false,
            // ADR-040 C2b scaffold (test fixture): the
            // `persistent_kv_cache` lift on `Qwen35LoadedModel` is iter-2b
            // scope; iter-2a always constructs it as `None` (per the
            // production `Qwen35LoadedModel::load` site).
            persistent_kv_cache: None,
        };
        let qwen = LoadedModel::Qwen35(qwen_loaded);

        // Accessor checks for the Qwen35 arm.
        assert_eq!(qwen.model_id(), "qwen-id");
        assert_eq!(qwen.context_length(), Some(1024));
        assert_eq!(qwen.quant_type(), Some("Q4_0"));
        assert_eq!(qwen.hidden_size(), 64);
        assert_eq!(qwen.vocab_size(), 256);
        assert_eq!(qwen.eos_token_ids(), &[151645]);
        assert_eq!(qwen.chat_template(), "qwen-template");
        assert_eq!(qwen.load_duration(), Duration::from_millis(7));
        assert!(
            qwen.prompt_cache().is_none(),
            "Qwen35 variant has no prompt_cache in iter-215 MVP"
        );
        // Tokenizer accessor returns a reference, no panic.
        let _ = qwen.tokenizer();
    }

    /// Phase B contract: when the GGUF lacks
    /// `tokenizer.ggml.eos_token_id`, `Qwen35LoadedModel::load`
    /// synthesizes the HF Qwen3.5 default (151645) per
    /// `cmd_generate_qwen35:1066-1069`.
    ///
    /// ADR-028 iter-267: extended to multi-EOS API.  The fallback path
    /// shape changed from `.unwrap_or(151645)` to an explicit
    /// `eos_token_ids.push(151_645)` after the multi-source scan
    /// (`tokenizer.ggml.eos_token_id` + `eot_token_id` + name-based
    /// scan of `tokenizer.ggml.tokens` for `<|im_end|>` /
    /// `<|endoftext|>`).  The behavior contract is preserved
    /// (151645 is still the final fallback when no source produced an
    /// EOS); the test pattern updates to match the iter-267 syntax.
    #[test]
    fn qwen35_loaded_model_load_synthesizes_eos_default_when_metadata_absent() {
        // The constant the constructor uses when the GGUF is silent.
        // Lifted to a local for clarity; if the constructor's
        // fallback drifts, this test fails until both are aligned.
        let expected_default: u32 = 151645;

        let src = include_str!("engine_qwen35.rs");
        assert!(
            src.contains("tokenizer.ggml.eos_token_id"),
            "Qwen35LoadedModel::load must read tokenizer.ggml.eos_token_id"
        );
        // Post-iter-267 contract: literal must appear inside an
        // empty-fallback push (the final fallback after multi-source
        // scan returns empty).  Allow either underscore-formatted or
        // plain literal.
        assert!(
            src.contains(&format!("push({expected_default})"))
                || src.contains(&format!(
                    "push({}_{})",
                    expected_default / 1000,
                    expected_default % 1000
                ))
                || src.contains(&format!("push(151_645)")),
            "Qwen35LoadedModel::load must default EOS to {expected_default} \
             (HF Qwen3.5 default per cmd_generate_qwen35) when the GGUF metadata \
             key is absent + name-scan finds no <|im_end|>/<|endoftext|>"
        );
    }

    /// Sanity: the iter-215 sentinel + message constants are
    /// non-empty and contain the operator-actionable literals
    /// (`hf2q generate` AND `cmd_generate_qwen35`).  The
    /// chat_completion 501 mapping at `handlers.rs` quotes the
    /// constants directly, so this test guards against a future
    /// maintainer trimming the constant body and breaking the
    /// operator contract.
    #[test]
    fn qwen35_not_implemented_message_names_both_workaround_literals() {
        let m = QWEN35_NOT_IMPLEMENTED_MESSAGE;
        assert!(!m.is_empty(), "message must be non-empty");
        assert!(
            m.contains("hf2q generate"),
            "501 message must name `hf2q generate` literal; got: {m}"
        );
        assert!(
            m.contains("cmd_generate_qwen35"),
            "501 message must name `cmd_generate_qwen35` literal; got: {m}"
        );
        let s = QWEN35_NOT_IMPLEMENTED_SENTINEL;
        assert!(!s.is_empty(), "sentinel must be non-empty");
    }

    // -----------------------------------------------------------------
    // Phase B-dense.2 follow-up — KV snapshot/restore worker bridge
    // -----------------------------------------------------------------
    //
    // Strategy: synthetic worker that owns an in-memory K/V byte map
    // keyed by (layer, slot). The worker handles KvSnapshot by reading
    // the in-memory bytes; KvRestore by writing them. This exercises
    // the request-reply round-trip on the real Engine surface without
    // a live Metal device or GGUF on disk. The "real-bytes" round-trip
    // discipline (per `feedback_substrate_must_not_synthesize_ship_gates`)
    // is satisfied: the worker reads/writes a real byte buffer and the
    // test asserts byte-equality at SHA-256 level.

    use std::collections::HashMap;

    /// Synthetic in-memory KV cache used by the test worker. Layered
    /// to mirror `MlxModelWeights.dense_kvs` shape:
    /// `cells[(layer, head, slot)] = head_dim bytes`.
    #[derive(Default)]
    struct SyntheticKvCache {
        nkv_per_layer: Vec<usize>,
        head_dim_per_layer: Vec<usize>,
        capacity_per_layer: Vec<usize>,
        is_sliding_per_layer: Vec<bool>,
        // Map from (layer, head, slot) -> [k_bytes, v_bytes]
        cells: HashMap<(usize, usize, usize), (Vec<u8>, Vec<u8>)>,
        write_pos_per_layer: Vec<u32>,
    }

    impl SyntheticKvCache {
        fn populate_layer(&mut self, layer: usize, seed: u8) {
            let nkv = self.nkv_per_layer[layer];
            let cap = self.capacity_per_layer[layer];
            let hd = self.head_dim_per_layer[layer];
            for h in 0..nkv {
                for slot in 0..cap {
                    let mut k = vec![0u8; hd];
                    let mut v = vec![0u8; hd];
                    for (i, b) in k.iter_mut().enumerate() {
                        *b = seed
                            ^ (layer as u8)
                            ^ (h as u8).wrapping_mul(7)
                            ^ (slot as u8).wrapping_mul(13)
                            ^ (i as u8).wrapping_mul(3)
                            ^ 0x5A;
                    }
                    for (i, b) in v.iter_mut().enumerate() {
                        *b = seed
                            ^ (layer as u8)
                            ^ (h as u8).wrapping_mul(7)
                            ^ (slot as u8).wrapping_mul(13)
                            ^ (i as u8).wrapping_mul(3)
                            ^ 0xA5;
                    }
                    self.cells.insert((layer, h, slot), (k, v));
                }
            }
        }
    }

    /// Build an Engine wrapping a synthetic worker that:
    /// - Holds a `SyntheticKvCache` in worker thread state.
    /// - Handles KvSnapshot by gathering bytes from `cells` in
    ///   token-position order over [range.start..range.end).
    /// - Handles KvRestore by writing bytes into `cells` from the
    ///   payloads.
    /// - Drops every other Request kind silently and exits on Shutdown.
    fn make_synthetic_kv_engine(
        nkv: Vec<usize>,
        head_dim: Vec<usize>,
        capacity: Vec<usize>,
        is_sliding: Vec<bool>,
        descriptor: Option<super::super::kv_spill_descriptor::KvSpillDescriptor>,
        seeded_layers: Vec<(usize, u8)>,
    ) -> Engine {
        let (tx, mut rx) = mpsc::channel::<Request>(8);
        let nkv_clone = nkv.clone();
        let hd_clone = head_dim.clone();
        let cap_clone = capacity.clone();
        let sliding_clone = is_sliding.clone();
        let handle = std::thread::Builder::new()
            .name("hf2q-engine-synthetic-kv".into())
            .spawn(move || {
                let mut cache = SyntheticKvCache {
                    nkv_per_layer: nkv_clone,
                    head_dim_per_layer: hd_clone,
                    capacity_per_layer: cap_clone,
                    is_sliding_per_layer: sliding_clone,
                    cells: HashMap::new(),
                    write_pos_per_layer: vec![0u32; nkv.len()],
                };
                for (layer, seed) in seeded_layers {
                    cache.populate_layer(layer, seed);
                }
                while let Some(req) = rx.blocking_recv() {
                    match req {
                        Request::Shutdown => break,
                        Request::KvSnapshot {
                            layer_rank,
                            range,
                            reply,
                        } => {
                            let result = if layer_rank >= cache.nkv_per_layer.len() {
                                Err(anyhow::anyhow!("test: layer OOB"))
                            } else {
                                let nkv = cache.nkv_per_layer[layer_rank];
                                let cap = cache.capacity_per_layer[layer_rank];
                                let hd = cache.head_dim_per_layer[layer_rank];
                                let is_sliding = cache.is_sliding_per_layer[layer_rank];
                                let n_tokens = (range.end - range.start) as usize;
                                let mut k_out = Vec::with_capacity(nkv * n_tokens * hd);
                                let mut v_out = Vec::with_capacity(nkv * n_tokens * hd);
                                let mut ok = true;
                                'gather: for h in 0..nkv {
                                    for tok in range.start..range.end {
                                        let slot = if is_sliding {
                                            (tok as usize) % cap
                                        } else {
                                            tok as usize
                                        };
                                        if !is_sliding && slot >= cap {
                                            ok = false;
                                            break 'gather;
                                        }
                                        match cache.cells.get(&(layer_rank, h, slot)) {
                                            Some((k, v)) => {
                                                k_out.extend_from_slice(k);
                                                v_out.extend_from_slice(v);
                                            }
                                            None => {
                                                k_out.extend_from_slice(&vec![0u8; hd]);
                                                v_out.extend_from_slice(&vec![0u8; hd]);
                                            }
                                        }
                                    }
                                }
                                if !ok {
                                    Err(anyhow::anyhow!("test: slot OOB"))
                                } else {
                                    Ok(Some(KvSnapshotBytes {
                                        k: k_out,
                                        v: v_out,
                                        nkv_heads: nkv as u16,
                                        head_dim: hd as u16,
                                        capacity: cap as u32,
                                        is_sliding,
                                        write_pos: if is_sliding {
                                            cache.write_pos_per_layer[layer_rank]
                                        } else {
                                            u32::MAX
                                        },
                                    }))
                                }
                            };
                            let _ = reply.send(result);
                        }
                        Request::KvRestore {
                            layer_rank,
                            range,
                            k_payload,
                            v_payload,
                            write_pos,
                            reply,
                        } => {
                            let result = if layer_rank >= cache.nkv_per_layer.len() {
                                Err(anyhow::anyhow!("test: layer OOB"))
                            } else {
                                let nkv = cache.nkv_per_layer[layer_rank];
                                let cap = cache.capacity_per_layer[layer_rank];
                                let hd = cache.head_dim_per_layer[layer_rank];
                                let is_sliding = cache.is_sliding_per_layer[layer_rank];
                                let n_tokens = (range.end - range.start) as usize;
                                let expected = nkv * n_tokens * hd;
                                if k_payload.len() != expected || v_payload.len() != expected {
                                    Err(anyhow::anyhow!("test: payload size mismatch"))
                                } else {
                                    let mut off = 0usize;
                                    for h in 0..nkv {
                                        for tok in range.start..range.end {
                                            let slot = if is_sliding {
                                                (tok as usize) % cap
                                            } else {
                                                tok as usize
                                            };
                                            cache.cells.insert(
                                                (layer_rank, h, slot),
                                                (
                                                    k_payload[off..off + hd].to_vec(),
                                                    v_payload[off..off + hd].to_vec(),
                                                ),
                                            );
                                            off += hd;
                                        }
                                    }
                                    if is_sliding && write_pos != u32::MAX {
                                        cache.write_pos_per_layer[layer_rank] = write_pos;
                                    }
                                    Ok(())
                                }
                            };
                            let _ = reply.send(result);
                        }
                        _ => {
                            // Ignore — test never awaits these.
                        }
                    }
                }
            })
            .expect("spawn synthetic kv worker");

        Engine {
            inner: Arc::new(EngineInner {
                tx,
                worker_handle: Mutex::new(Some(handle)),
                info: synthetic_load_info("synth-kv"),
                arch: LoadedArch::Gemma,
                model_id: "synth-kv".into(),
                context_length: None,
                quant_type: None,
                hidden_size: 0,
                vocab_size: 0,
                eos_token_ids: vec![],
                tokenizer: Arc::new(Tokenizer::new(tokenizers::models::bpe::BPE::default())),
                chat_template: Arc::new(String::new()),
                registration: None,
                token_bytes: std::sync::OnceLock::new(),
                kv_spill_descriptor: descriptor,
                tq_packed_descriptor: None,
                mode: EngineMode::SerialFifo,
                // ADR-040 C2b scaffold for synthetic test fixtures.
                max_slots: 1,
                // ADR-040 §3.5 iter-A5b: synthetic fixtures opt out.
                per_slot_kv_budget_bytes: 0,
                kv_bytes_per_token_cached: 0,
                scheduler_stats_snapshot: Arc::new(Mutex::new(SchedulerStats {
                    policy: SchedulerPolicy::FifoSerial,
                    in_flight_slots: 0,
                    queue_capacity: 8,
                    admitted_total: 0,
                    rejected_429_total: 0,
                    completed_total: 0,
                })),
            }),
        }
    }

    /// Test request_kv_1: kv_spill_descriptor returns Some for an
    /// engine constructed with one. Falsifier: Engine::spawn would
    /// have to leave the field as None even when given a Gemma model.
    #[test]
    fn request_kv_descriptor_returns_some_when_set() {
        use super::super::kv_spill_descriptor::{KvDType, KvSpillDescriptor};
        use crate::serve::config::LayerType;
        let d = KvSpillDescriptor {
            sliding_window: 16,
            max_decode_tokens: 32,
            num_layers: 2,
            layer_types: vec![LayerType::Sliding, LayerType::Full],
            nkv_heads: vec![2, 1],
            head_dim: vec![8, 16],
            kv_dtype: KvDType::F32,
            provenance: super::super::kv_spill_descriptor::KvSpillProvenance::default(),
        };
        let engine = make_synthetic_kv_engine(
            vec![2, 1],
            vec![8, 16],
            vec![16, 32],
            vec![true, false],
            Some(d.clone()),
            vec![],
        );
        let got = engine.kv_spill_descriptor().expect("Some");
        assert_eq!(got.sliding_window, 16);
        assert_eq!(got.num_layers, 2);
        assert_eq!(got.layer_types[0], LayerType::Sliding);
        assert_eq!(got.layer_types[1], LayerType::Full);
        assert_eq!(got.nkv_heads, vec![2, 1]);
        assert_eq!(got.head_dim, vec![8, 16]);
    }

    /// Test request_kv_2: kv_spill_descriptor returns None when
    /// not set (matches Qwen35 path). Falsifier: descriptor leaks
    /// across architectures.
    #[test]
    fn request_kv_descriptor_returns_none_when_not_set() {
        let engine = make_synthetic_kv_engine(vec![1], vec![4], vec![8], vec![true], None, vec![]);
        assert!(engine.kv_spill_descriptor().is_none());
    }

    /// Test request_kv_3: request_kv_snapshot round-trips real bytes
    /// from a populated synthetic cache. Falsifier: returned bytes
    /// don't match the seed pattern.
    #[test]
    fn request_kv_snapshot_returns_real_bytes_from_populated_layer() {
        let engine = make_synthetic_kv_engine(
            vec![2],
            vec![4],
            vec![8],
            vec![true],
            None,
            vec![(0usize, 0x42u8)],
        );
        let got = engine
            .request_kv_snapshot(0, 0..4)
            .expect("snapshot ok")
            .expect("populated layer ⇒ Some");
        assert_eq!(got.nkv_heads, 2);
        assert_eq!(got.head_dim, 4);
        assert_eq!(got.capacity, 8);
        assert!(got.is_sliding);
        // 2 heads * 4 tokens * 4 head_dim = 32 bytes per K and V.
        assert_eq!(got.k.len(), 32);
        assert_eq!(got.v.len(), 32);
        // Verify the seed pattern: cell (layer=0, head=0, slot=0)
        // first byte should be 0x42 ^ 0 ^ 0 ^ 0 ^ 0 ^ 0x5A = 0x18.
        let expected_first_k = 0x42u8 ^ 0u8 ^ 0u8 ^ 0u8 ^ 0u8 ^ 0x5Au8;
        assert_eq!(
            got.k[0], expected_first_k,
            "first K byte matches seed pattern"
        );
        let expected_first_v = 0x42u8 ^ 0u8 ^ 0u8 ^ 0u8 ^ 0u8 ^ 0xA5u8;
        assert_eq!(
            got.v[0], expected_first_v,
            "first V byte matches seed pattern"
        );
    }

    /// Test request_kv_4: request_kv_snapshot returns Ok(None) for
    /// an unpopulated synthetic cache (layers exist but no cells).
    /// Falsifier: returns spurious zero bytes instead of None.
    ///
    /// (Per the synthetic worker contract: layers without seeded
    /// cells fill with zeros — but the snapshot still succeeds. The
    /// "no prefill yet" case is modelled by a layer with capacity=0
    /// in the synthetic harness.)
    #[test]
    fn request_kv_snapshot_zero_capacity_layer_returns_none_path() {
        // Layer with capacity=0 simulates "no prefill yet" — the
        // production worker returns Ok(None) when dense_kvs is None.
        // Our synthetic worker doesn't model that exact path, but a
        // capacity-0 layer triggers an error in gather (not OOB —
        // n_tokens=0). Use range collapse to cover Ok(None) at the
        // descriptor level instead: descriptor.num_layers > engine
        // layer count = layer OOB error.
        let engine = make_synthetic_kv_engine(vec![1], vec![4], vec![8], vec![true], None, vec![]);
        // Layer 99 doesn't exist — synthetic worker returns Err.
        let err = engine.request_kv_snapshot(99, 0..4);
        assert!(err.is_err(), "out-of-range layer ⇒ Err from worker");
    }

    /// Test request_kv_5: request_kv_restore + request_kv_snapshot
    /// round-trip is byte-exact. Load-bearing for the H4 hypothesis.
    /// Falsifier: any byte mismatch.
    #[test]
    fn request_kv_restore_then_snapshot_round_trip_byte_exact() {
        let engine = make_synthetic_kv_engine(vec![1], vec![4], vec![8], vec![true], None, vec![]);
        // Build deterministic payload: 1 head * 4 tokens * 4 head_dim
        // = 16 bytes per K and V.
        let k_payload: Vec<u8> = (0..16u8).map(|i| i.wrapping_mul(7) ^ 0xC3).collect();
        let v_payload: Vec<u8> = (0..16u8).map(|i| i.wrapping_mul(11) ^ 0x3C).collect();
        engine
            .request_kv_restore(0, 0..4, k_payload.clone(), v_payload.clone(), 3)
            .expect("restore ok");
        // Snapshot back.
        let got = engine
            .request_kv_snapshot(0, 0..4)
            .expect("snapshot ok")
            .expect("Some after restore");
        assert_eq!(got.k, k_payload, "K bytes round-trip exact");
        assert_eq!(got.v, v_payload, "V bytes round-trip exact");
        assert_eq!(got.write_pos, 3, "sliding write_pos preserved");
    }

    /// Test request_kv_6: request_kv_restore returns Err on shape
    /// mismatch. Falsifier: silently truncates.
    #[test]
    fn request_kv_restore_shape_mismatch_returns_err() {
        let engine = make_synthetic_kv_engine(vec![1], vec![4], vec![8], vec![true], None, vec![]);
        // Range expects 4 tokens * 1 head * 4 head_dim = 16 bytes.
        // Pass only 4 bytes — must fail.
        let bad = vec![0u8; 4];
        let result = engine.request_kv_restore(0, 0..4, bad.clone(), bad, u32::MAX);
        assert!(
            result.is_err(),
            "payload shape mismatch must surface as Err"
        );
    }

    // ---------------------------------------------------------------------------
    // ADR-040 Phase C iter-2a — byte-equivalence regression pin
    // (dossier §2.5 + §4 iter-2a step 1)
    //
    // Construct two engines on identical inputs — engine_a via the 3-arg
    // pre-ADR-040 `Engine::spawn` entry point and engine_b via the
    // iter-1.5 `Engine::spawn_with_mode(..., EngineMode::SerialFifo)`
    // entry point — drive the same greedy prompt through both, and assert
    // byte-equality on every observable `GenerationResult` field that is
    // NOT timing-derived. ADR-040 §3.6's load-bearing pledge is that
    // `FifoSerial` is bit-equivalent to pre-ADR-040; this test is the
    // load-bearing falsifier for that pledge.
    //
    // ---------------------------------------------------------------------------
    // Env-gating: HF2Q_BYTE_EQUIV_E2E=1 + HF2Q_BYTE_EQUIV_E2E_GGUF=<path>
    //
    // The existing `make_synthetic_kv_engine_for_test` fixture at
    // engine.rs:603 spawns a synthetic worker that drains the channel
    // WITHOUT running real `generate_once` inference (the dossier §2.10
    // calls this out explicitly: "useful for the `EngineInner` lifecycle
    // / handler-route tests but NOT for the byte-equivalence test which
    // needs real `generate_once` execution"). The dossier R8 mitigation
    // is therefore an env-gated real-GGUF path; when the env is absent
    // the test prints a skip notice and passes trivially. Hot-loop CI
    // and developer-laptop `cargo test` runs (no GGUF on disk, no env
    // set) do not pay the load cost; the regression-pin runs under
    // explicit operator invocation:
    //
    //   HF2Q_BYTE_EQUIV_E2E=1 \
    //   HF2Q_BYTE_EQUIV_E2E_GGUF=/path/to/tiny.gguf \
    //   cargo test --release --bin hf2q -- \
    //     engine_serial_fifo_byte_equivalent_to_pre_phase_c
    //
    // Mirrors the env-gating pattern at tests/multi_model_swap.rs:93-103.
    //
    // ---------------------------------------------------------------------------
    // What the test asserts (per ADR-040 §3.6 + §2.5 of the dossier):
    //
    //   - `text` (the rendered completion text)
    //   - `prompt_tokens` (usage counter)
    //   - `completion_tokens` (usage counter)
    //   - `reasoning_tokens` (usage counter)
    //   - `cached_tokens` (usage counter)
    //   - `finish_reason` ("stop" | "length")
    //   - `reasoning_text` (Option<String>)
    //   - `logprobs` (Option<Vec<f32>>)
    //
    // What the test EXCLUDES (timing / non-determinism):
    //
    //   - `prefill_duration` / `decode_duration` (wall-clock, not
    //     byte-comparable)
    //
    // `GenerationResult` is `#[derive(Debug, Clone)]` (not `PartialEq`);
    // the test asserts field-by-field via `assert_eq!`, which keeps the
    // pin readable on a divergence and avoids a derive change to the
    // public production type.
    //
    // Vacuous-test guard: the test rejects an empty completion (`text`
    // empty AND `completion_tokens == 0`) — without this, the
    // byte-equality assertions are trivially true on a fixture that
    // silently produces no output.
    //
    // ---------------------------------------------------------------------------
    // Why this is the C2b regression-pin foundation:
    //
    // C2a (this iter) ships the test FIRST against HEAD, where
    // `spawn_with_mode(SerialFifo)` already delegates to `spawn`
    // (iter-1.5 F1 at engine.rs:2636-2662). The test PASSES today
    // because both engines hit identical `worker_run` code. C2b refactors
    // `worker_run` to thread a `Box<dyn Scheduler>` through the dispatch
    // arms — and this test FALSIFIES on any wrapper that mutates the
    // observable output (off-by-one in `advance_after_decode`, a token
    // dropped because `step()` returned `Idle` prematurely, sampler RNG
    // seed shift from inserting a tokio-runtime layer between
    // `try_send` and the worker). Per dossier §2.5: "Catches: any
    // worker-loop wrapper that mutates state."
    // ---------------------------------------------------------------------------

    const BYTE_EQUIV_E2E_ENV_GATE: &str = "HF2Q_BYTE_EQUIV_E2E";
    const BYTE_EQUIV_E2E_GGUF_ENV: &str = "HF2Q_BYTE_EQUIV_E2E_GGUF";

    /// Returns `true` if the test should skip (env not gated). When `true`
    /// the caller has already emitted a skip notice via `eprintln!`.
    fn byte_equiv_skip_unless_gated(test_name: &str) -> bool {
        if std::env::var(BYTE_EQUIV_E2E_ENV_GATE).as_deref() == Ok("1") {
            return false;
        }
        eprintln!(
            "[skip] {test_name} — set {BYTE_EQUIV_E2E_ENV_GATE}=1 + \
             {BYTE_EQUIV_E2E_GGUF_ENV}=<path> to run the ADR-040 C2a \
             byte-equivalence regression pin. Dossier §2.5 + §4 iter-2a \
             step 1; mitigates R8 (synthetic fixture cannot exercise \
             real generate_once)."
        );
        true
    }

    #[test]
    fn engine_serial_fifo_byte_equivalent_to_pre_phase_c() {
        if byte_equiv_skip_unless_gated("engine_serial_fifo_byte_equivalent_to_pre_phase_c") {
            return;
        }

        let gguf_path: PathBuf = std::env::var(BYTE_EQUIV_E2E_GGUF_ENV)
            .map(PathBuf::from)
            .unwrap_or_else(|_| {
                panic!(
                    "ADR-040 C2a: {BYTE_EQUIV_E2E_ENV_GATE}=1 set without \
                     {BYTE_EQUIV_E2E_GGUF_ENV}=<path>. The byte-equivalence \
                     pin needs a real GGUF on disk to run `generate_once` \
                     against both spawn entry points; the synthetic fixture \
                     cannot serve this role (dossier §2.10 + R8)."
                )
            });
        assert!(
            gguf_path.exists(),
            "ADR-040 C2a: {BYTE_EQUIV_E2E_GGUF_ENV} points to a missing \
             file: {}. Set it to a valid GGUF path.",
            gguf_path.display()
        );

        // Build TWO independent `LoadedModel` instances from the SAME
        // GGUF byte source. Two separate `LoadedModel::load` calls (not
        // a `.clone()` — `LoadedModel` does not derive Clone, and the
        // C2a contract is about whether the two SPAWN entry points
        // produce byte-equivalent output on *equivalent* inputs, not
        // whether one in-memory model can drive both engines).
        let load_opts = LoadOptions {
            model_path: gguf_path.clone(),
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };
        let loaded_a = LoadedModel::load(&load_opts).expect("LoadedModel::load (a)");
        let loaded_b = LoadedModel::load(&load_opts).expect("LoadedModel::load (b)");

        // Identical queue capacity + identical KV budget (None) for both
        // engines. The two spawn entry points MUST converge on identical
        // `worker_run` behaviour at HEAD per iter-1.5 F1 (engine.rs:2636-
        // 2662 — `EngineMode::SerialFifo` delegates to 3-arg `spawn`).
        let queue_capacity: usize = 4;
        let kv_cache_budget_bytes: Option<u64> = None;

        let engine_a = Engine::spawn(loaded_a, queue_capacity, kv_cache_budget_bytes);
        let engine_b = Engine::spawn_with_mode(
            loaded_b,
            queue_capacity,
            kv_cache_budget_bytes,
            EngineMode::SerialFifo,
        )
        .expect(
            "ADR-040 iter-1.5 F1: EngineMode::SerialFifo MUST succeed at \
             spawn_with_mode (it delegates to 3-arg spawn)",
        );

        // Greedy prompt — temperature=0.0 is deterministic regardless of
        // RNG seed (per SamplingParams docs at engine.rs:273-274
        // "Greedy (T=0) decodes are deterministic regardless"). max_tokens
        // small to keep the pin fast under E2E mode; large enough to
        // catch a mid-decode wrapper bug.
        let prompt_tokens: Vec<u32> = vec![1u32, 2, 3, 4, 5];
        let params = SamplingParams {
            temperature: 0.0,
            max_tokens: 16,
            ..Default::default()
        };

        let rt = tokio::runtime::Builder::new_current_thread()
            .enable_all()
            .build()
            .expect("build current-thread tokio runtime");

        let result_a = rt
            .block_on(engine_a.generate(prompt_tokens.clone(), params.clone()))
            .expect("engine_a generate (3-arg spawn path)");
        let result_b = rt
            .block_on(engine_b.generate(prompt_tokens, params))
            .expect("engine_b generate (spawn_with_mode SerialFifo path)");

        // Vacuous-test guard: if the fixture silently produced no
        // output, the byte-equality assertions below are trivially true
        // on empty/zero values. Reject before any field comparison.
        assert!(
            !result_a.text.is_empty() || result_a.completion_tokens > 0,
            "ADR-040 C2a vacuous test: engine_a produced empty text AND \
             zero completion_tokens (text={:?}, completion_tokens={}) — \
             the synthetic prompt did not exercise real decode; \
             byte-equality below would pass trivially. Use a non-trivial \
             prompt or a fixture with deterministic non-empty output.",
            result_a.text,
            result_a.completion_tokens,
        );

        // Field-by-field byte equality. GenerationResult is not
        // PartialEq; field-wise asserts give a precise failure surface
        // on divergence + avoid mutating the public derive set.
        // Timing fields (prefill_duration, decode_duration) are
        // intentionally excluded — wall-clock varies run-to-run.
        assert_eq!(
            result_a.text, result_b.text,
            "ADR-040 C2a byte-equivalence FALSIFIED: FifoSerial `text` \
             differs from pre-C2 path. spawn_with_mode(SerialFifo) MUST \
             produce byte-identical decoded text to 3-arg spawn at HEAD."
        );
        assert_eq!(
            result_a.reasoning_text, result_b.reasoning_text,
            "ADR-040 C2a byte-equivalence FALSIFIED: FifoSerial \
             `reasoning_text` differs from pre-C2 path."
        );
        assert_eq!(
            result_a.prompt_tokens, result_b.prompt_tokens,
            "ADR-040 C2a byte-equivalence FALSIFIED: FifoSerial \
             `prompt_tokens` counter differs from pre-C2 path."
        );
        assert_eq!(
            result_a.completion_tokens, result_b.completion_tokens,
            "ADR-040 C2a byte-equivalence FALSIFIED: FifoSerial \
             `completion_tokens` counter differs from pre-C2 path."
        );
        assert_eq!(
            result_a.reasoning_tokens, result_b.reasoning_tokens,
            "ADR-040 C2a byte-equivalence FALSIFIED: FifoSerial \
             `reasoning_tokens` counter differs from pre-C2 path."
        );
        assert_eq!(
            result_a.cached_tokens, result_b.cached_tokens,
            "ADR-040 C2a byte-equivalence FALSIFIED: FifoSerial \
             `cached_tokens` counter differs from pre-C2 path."
        );
        assert_eq!(
            result_a.finish_reason, result_b.finish_reason,
            "ADR-040 C2a byte-equivalence FALSIFIED: FifoSerial \
             `finish_reason` differs from pre-C2 path."
        );
        assert_eq!(
            result_a.logprobs, result_b.logprobs,
            "ADR-040 C2a byte-equivalence FALSIFIED: FifoSerial \
             `logprobs` vector differs from pre-C2 path."
        );

        // Drain the worker threads so the test exits cleanly (the
        // worker_run thread holds the LoadedModel — leaving it running
        // would leak GPU buffers across test cases).
        rt.block_on(engine_a.shutdown()).expect("engine_a shutdown");
        rt.block_on(engine_b.shutdown()).expect("engine_b shutdown");
    }

    // =======================================================================
    // ADR-040 Phase F M1 (F1) — SlotAware batched-worker-loop proof pins.
    //
    // These reuse the byte-equiv env gate (HF2Q_BYTE_EQUIV_E2E=1 +
    // HF2Q_BYTE_EQUIV_E2E_GGUF=<path>) — they need a real GGUF because the
    // proof is about real per-slot KV + sampling, which the synthetic
    // fixture cannot exercise. They are the H-M1 falsifiers from §0.12:
    // any slot diverging from its serial reference, or N=1 regressing,
    // fails the milestone.
    // =======================================================================

    /// Compare two `GenerationResult`s field-by-field (excluding wall-clock
    /// timing). `ctx` names the comparison for the failure surface.
    fn assert_genresult_byte_equal(a: &GenerationResult, b: &GenerationResult, ctx: &str) {
        assert_eq!(a.text, b.text, "{ctx}: `text` diverged");
        assert_eq!(
            a.reasoning_text, b.reasoning_text,
            "{ctx}: `reasoning_text` diverged"
        );
        assert_eq!(
            a.prompt_tokens, b.prompt_tokens,
            "{ctx}: `prompt_tokens` diverged"
        );
        assert_eq!(
            a.completion_tokens, b.completion_tokens,
            "{ctx}: `completion_tokens` diverged"
        );
        assert_eq!(
            a.reasoning_tokens, b.reasoning_tokens,
            "{ctx}: `reasoning_tokens` diverged"
        );
        assert_eq!(
            a.cached_tokens, b.cached_tokens,
            "{ctx}: `cached_tokens` diverged"
        );
        assert_eq!(
            a.finish_reason, b.finish_reason,
            "{ctx}: `finish_reason` diverged"
        );
        assert_eq!(a.logprobs, b.logprobs, "{ctx}: `logprobs` diverged");
    }

    /// Compute the SERIAL slot-aware reference for one gemma4 prompt: load a
    /// fresh model, provision the multi-seq KV at n_seqs=1, and run the
    /// existing `generate_gemma4_once_slot_aware` inline at SlotId(0). This
    /// is the AC4-correct bar for F1's batched path (SAME forward path as
    /// the SlotAware loop), decoupled from the pre-existing legacy
    /// `generate_once`-vs-slot-aware-forward delta pinned by h77. Each call
    /// uses its own model so references are independent.
    fn gemma4_serial_slot_aware_ref(
        load_opts: &LoadOptions,
        prompt: &[u32],
        params: &SamplingParams,
    ) -> GenerationResult {
        gemma4_serial_slot_aware_ref_at(load_opts, prompt, params, 1, SlotId(0))
    }

    /// Run the serial slot-aware generate at a SPECIFIC slot_id with a
    /// SPECIFIC n_seqs provisioning — a single request, no concurrency. Pins
    /// per-slot KV byte-offset indexing in ISOLATION.
    fn gemma4_serial_slot_aware_ref_at(
        load_opts: &LoadOptions,
        prompt: &[u32],
        params: &SamplingParams,
        n_seqs: u32,
        slot_id: SlotId,
    ) -> GenerationResult {
        let mut loaded = LoadedModel::load(load_opts).expect("load ref model");
        let LoadedModel::Gemma(g) = &mut loaded else {
            panic!("gemma4_serial_slot_aware_ref: expected a Gemma GGUF")
        };
        g.provision_multi_seq_kv_for_slot_aware(n_seqs)
            .expect("provision multi-seq KV");
        let mut kv = g.multi_seq_kv.take().expect("multi_seq_kv provisioned");
        let mut hybrid = g.multi_seq_kv_hybrid.take();
        let mut dense = g.multi_seq_kv_dense.take();
        let mut mlx = g.multi_seq_kv_mlx.take();
        let r = generate_gemma4_once_slot_aware(
            g,
            prompt,
            params,
            None,
            &mut kv,
            hybrid.as_mut(),
            dense.as_mut(),
            mlx.as_mut(),
            slot_id,
        )
        .expect("serial slot-aware ref");
        g.multi_seq_kv = Some(kv);
        g.multi_seq_kv_hybrid = hybrid;
        g.multi_seq_kv_dense = dense;
        g.multi_seq_kv_mlx = mlx;
        r
    }

    /// ADR-040 §0.12 STEP 1b golden-output table — captured pre-refactor
    /// 2026-06-24 (gemma4 Q5_K_M, hybrid TQ-8, T=0, the four fixed prompts
    /// in `slot_aware_serial_golden_output_pin`).  Hoisted to module scope
    /// per the H1 structural audit (`tests/structural_audit_serve_consts.rs`):
    /// fn-body `const` is fn-local-scope and therefore unreachable from
    /// sibling test modules — policy data belongs at module scope.
    const GOLDEN_OUTPUT: &[&str] = &[
        "</i></p>\n<p>\n<style>\n/* Global Styles",
        "텐츠텐츠텐츠텐츠텐츠를앞의의를를를를를맞는",
        "________________________________________________________________________________________________________________________________________________________________",
        "\\|_{**}**\n\n---\n\n## 1. Introduction\nThe purpose of",
    ];

    /// ADR-040 §0.12 STEP 1b GUARDRAIL (golden-output pin) — the gemma4
    /// stateless-forward refactor moves per-request KV cursor state from
    /// shared self.kv_caches into the per-slot scaffold; it must change
    /// STORAGE LOCATION, NOT NUMERICS. This pin asserts the serial
    /// slot-aware path (generate_gemma4_once_slot_aware) produces the SAME
    /// text on a fixed prompt set at T=0 after the refactor as before
    /// (captured pre-refactor 2026-06-24). If this RED's after the
    /// refactor, the refactor changed numerics — a regression, stop.
    ///
    /// Golden values captured pre-refactor on the serve.sh gemma4 Q5_K_M
    /// (hybrid TQ-8 default). Greedy (T=0) → deterministic.
    #[test]
    fn slot_aware_serial_golden_output_pin() {
        if byte_equiv_skip_unless_gated("slot_aware_serial_golden_output_pin") {
            return;
        }
        let gguf_path: PathBuf = std::env::var(BYTE_EQUIV_E2E_GGUF_ENV)
            .map(PathBuf::from)
            .expect("HF2Q_BYTE_EQUIV_E2E_GGUF set");
        assert!(gguf_path.exists(), "GGUF missing: {}", gguf_path.display());
        let load_opts = LoadOptions {
            model_path: gguf_path.clone(),
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };
        // Fixed prompt set (the same prompts the N=4 parity + interleave
        // tests use, so golden ↔ parity are directly comparable).
        let prompts: Vec<Vec<u32>> = vec![
            vec![1u32, 2, 3, 4, 5],
            vec![10u32, 11, 12, 13],
            vec![8u32, 9],
            vec![2u32, 4, 6, 8],
        ];
        let params = SamplingParams {
            temperature: 0.0,
            max_tokens: 16,
            ..Default::default()
        };
        let mut golden: Vec<String> = Vec::new();
        for p in &prompts {
            let r = gemma4_serial_slot_aware_ref(&load_opts, p, &params);
            golden.push(r.text.clone());
            eprintln!(
                "[golden] prompt={:?} completion_tokens={} text={:?}",
                p, r.completion_tokens, r.text
            );
        }
        // GOLDEN_OUTPUT (module scope, see above): filled in from the
        // eprintln on the pre-refactor capture run, then this block
        // asserts. Until populated (capture run), the eprintln above is
        // the capture; the assert below is the post-refactor regression
        // guard. Captured pre-refactor 2026-06-24 (gemma4 Q5_K_M, hybrid
        // TQ-8, T=0).
        for (i, (got, want)) in golden.iter().zip(GOLDEN_OUTPUT.iter()).enumerate() {
            assert_eq!(
                got, want,
                "ADR-040 STEP 1b golden pin FALSIFIED at prompt {i}: serial \
                 slot-aware output changed across the stateless refactor \
                 (numerics regressed, not just storage location)."
            );
        }
    }

    /// ADR-040 §0.12 B2 AUDIT — per-slot KV byte-offset isolation, NO
    /// concurrency. Same prompt as a SINGLE request at SlotId(0)/(1)/(3)
    /// (n_seqs=4) must all == the SlotId(0)/n_seqs=1 ref. A slot k>0
    /// divergence ⇒ per-slot view byte-offset bug (KV-layout, not
    /// concurrency). All-match ⇒ per-slot indexing correct; N>1 divergence
    /// is purely interleave/shared-state.
    #[test]
    fn slot_aware_per_slot_kv_offset_isolation() {
        if byte_equiv_skip_unless_gated("slot_aware_per_slot_kv_offset_isolation") {
            return;
        }
        let gguf_path: PathBuf = std::env::var(BYTE_EQUIV_E2E_GGUF_ENV)
            .map(PathBuf::from)
            .expect("HF2Q_BYTE_EQUIV_E2E_GGUF set");
        assert!(gguf_path.exists(), "GGUF missing: {}", gguf_path.display());
        let load_opts = LoadOptions {
            model_path: gguf_path.clone(),
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };
        let prompt: Vec<u32> = vec![1u32, 2, 3, 4, 5];
        let params = SamplingParams {
            temperature: 0.0,
            max_tokens: 12,
            ..Default::default()
        };

        let r_ref = gemma4_serial_slot_aware_ref_at(&load_opts, &prompt, &params, 1, SlotId(0));
        for k in [0u32, 1, 3] {
            let r_k = gemma4_serial_slot_aware_ref_at(&load_opts, &prompt, &params, 4, SlotId(k));
            assert_genresult_byte_equal(
                &r_k,
                &r_ref,
                &format!("ADR-040 B2 — single-request SlotId({k})/n_seqs=4 vs SlotId(0)/n_seqs=1"),
            );
        }
    }

    /// ADR-040 §0.12 B2 AUDIT — deterministic INTERLEAVE reproduction (no
    /// tokio, no scheduler). Mimics the F1 loop's order on ONE model with
    /// n_seqs=2: prefill slot0, prefill slot1, then alternate decode
    /// slot0/slot1 for several steps. Captures each slot's greedy token
    /// stream and compares to that slot's ATOMIC serial reference (prefill+
    /// full-decode with no interleave). If a slot's interleaved stream
    /// diverges from its atomic stream, the leak is shared self.* state
    /// corrupted by the OTHER slot's interleaved forward (the residual B2
    /// bug). Pinpoints exactly which step diverges. Runs through the raw
    /// forward primitives so it's deterministic + debuggable.
    #[test]
    fn slot_aware_interleave_two_slots_vs_atomic() {
        if byte_equiv_skip_unless_gated("slot_aware_interleave_two_slots_vs_atomic") {
            return;
        }
        let gguf_path: PathBuf = std::env::var(BYTE_EQUIV_E2E_GGUF_ENV)
            .map(PathBuf::from)
            .expect("HF2Q_BYTE_EQUIV_E2E_GGUF set");
        assert!(gguf_path.exists(), "GGUF missing: {}", gguf_path.display());
        let load_opts = LoadOptions {
            model_path: gguf_path.clone(),
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };
        // Two DISTINCT prompts so cross-slot contamination is visible.
        let p0: Vec<u32> = vec![1u32, 2, 3, 4, 5];
        let p1: Vec<u32> = vec![10u32, 11, 12, 13];
        let n_dec = 10usize;

        // Atomic per-slot reference token streams (no interleave).
        let atomic_stream = |prompt: &[u32]| -> Vec<u32> {
            let mut m = LoadedModel::load(&load_opts).expect("load atomic");
            let LoadedModel::Gemma(g) = &mut m else {
                panic!("Gemma")
            };
            g.provision_multi_seq_kv_for_slot_aware(1).expect("prov");
            let mut kv = g.multi_seq_kv.take().unwrap();
            let mut hyb = g.multi_seq_kv_hybrid.take();
            let mut den = g.multi_seq_kv_dense.take();
            let mut mlx = g.multi_seq_kv_mlx.take();
            let mut toks = Vec::new();
            let first = g
                .weights
                .forward_prefill_with_soft_tokens_slot_aware(
                    prompt,
                    &[],
                    n_dec,
                    &mut g.ctx,
                    SlotId(0),
                    &mut kv,
                    hyb.as_mut(),
                    den.as_mut(),
                    mlx.as_mut(),
                )
                .expect("atomic prefill");
            toks.push(first);
            let mut feed = first;
            for step in 1..n_dec {
                let pos = prompt.len() + step - 1;
                let mut pr: Option<crate::inference::models::gemma4::profile::TokenProfile> = None;
                let t = g
                    .weights
                    .forward_decode_slot_aware(
                        feed,
                        pos,
                        &mut g.ctx,
                        &mut pr,
                        SlotId(0),
                        &mut kv,
                        hyb.as_mut(),
                        den.as_mut(),
                        mlx.as_mut(),
                    )
                    .expect("atomic decode");
                toks.push(t);
                feed = t;
            }
            toks
        };
        let ref0 = atomic_stream(&p0);
        let ref1 = atomic_stream(&p1);
        assert_ne!(ref0, ref1, "vacuous: distinct prompts gave same stream");

        // Interleaved run on ONE model, n_seqs=2, slot0=p0 slot1=p1.
        let mut m = LoadedModel::load(&load_opts).expect("load interleave");
        let LoadedModel::Gemma(g) = &mut m else {
            panic!("Gemma")
        };
        g.provision_multi_seq_kv_for_slot_aware(2).expect("prov2");
        let mut kv = g.multi_seq_kv.take().unwrap();
        let mut hyb = g.multi_seq_kv_hybrid.take();
        let mut den = g.multi_seq_kv_dense.take();
        let mut mlx = g.multi_seq_kv_mlx.take();
        // Mimic clear_gemma4_self_mounts before each prefill.
        let clear = |g: &mut GemmaLoadedModel| {
            g.weights.dense_kvs = None;
            g.weights.hybrid_kv = None;
            g.weights.leg_hb_encoded = None;
        };
        clear(g);
        let f0 = g
            .weights
            .forward_prefill_with_soft_tokens_slot_aware(
                &p0,
                &[],
                n_dec,
                &mut g.ctx,
                SlotId(0),
                &mut kv,
                hyb.as_mut(),
                den.as_mut(),
                mlx.as_mut(),
            )
            .expect("il prefill0");
        clear(g);
        let f1 = g
            .weights
            .forward_prefill_with_soft_tokens_slot_aware(
                &p1,
                &[],
                n_dec,
                &mut g.ctx,
                SlotId(1),
                &mut kv,
                hyb.as_mut(),
                den.as_mut(),
                mlx.as_mut(),
            )
            .expect("il prefill1");
        let mut s0 = vec![f0];
        let mut s1 = vec![f1];
        let (mut feed0, mut feed1) = (f0, f1);
        for step in 1..n_dec {
            let pos0 = p0.len() + step - 1;
            let mut pr0: Option<crate::inference::models::gemma4::profile::TokenProfile> = None;
            let t0 = g
                .weights
                .forward_decode_slot_aware(
                    feed0,
                    pos0,
                    &mut g.ctx,
                    &mut pr0,
                    SlotId(0),
                    &mut kv,
                    hyb.as_mut(),
                    den.as_mut(),
                    mlx.as_mut(),
                )
                .expect("il decode0");
            s0.push(t0);
            feed0 = t0;
            let pos1 = p1.len() + step - 1;
            let mut pr1: Option<crate::inference::models::gemma4::profile::TokenProfile> = None;
            let t1 = g
                .weights
                .forward_decode_slot_aware(
                    feed1,
                    pos1,
                    &mut g.ctx,
                    &mut pr1,
                    SlotId(1),
                    &mut kv,
                    hyb.as_mut(),
                    den.as_mut(),
                    mlx.as_mut(),
                )
                .expect("il decode1");
            s1.push(t1);
            feed1 = t1;
        }
        g.multi_seq_kv = Some(kv);
        g.multi_seq_kv_hybrid = hyb;
        g.multi_seq_kv_dense = den;
        g.multi_seq_kv_mlx = mlx;

        let first_diff = |a: &[u32], b: &[u32]| -> Option<usize> {
            a.iter().zip(b.iter()).position(|(x, y)| x != y)
        };
        eprintln!(
            "[B2-interleave] slot0 interleaved-vs-atomic first_diff={:?} (atomic={:?} il={:?})",
            first_diff(&s0, &ref0),
            ref0,
            s0
        );
        eprintln!(
            "[B2-interleave] slot1 interleaved-vs-atomic first_diff={:?} (atomic={:?} il={:?})",
            first_diff(&s1, &ref1),
            ref1,
            s1
        );
        assert_eq!(
            s0, ref0,
            "ADR-040 B2 — slot0 interleaved stream diverged from atomic"
        );
        assert_eq!(
            s1, ref1,
            "ADR-040 B2 — slot1 interleaved stream diverged from atomic"
        );
    }

    /// F1 AC4 (ADR-040 §0.12, ruling (b) 2026-06-24) — SlotAware at N=1 is
    /// byte-identical to the SERIAL SLOT-AWARE reference
    /// (`generate_gemma4_once_slot_aware`), i.e. the SAME forward path.
    ///
    /// This is F1's correctness bar: F1 changes the ORCHESTRATION (a
    /// scheduler-driven, admit-while-decoding loop) over the slot-aware
    /// forward, so the honest pin is that the loop drives that forward
    /// faithfully — byte-identical to running the serial slot-aware fn
    /// inline at N=1. It is NOT compared to SerialFifo: gemma4 SerialFifo
    /// routes SlotId(0) through the LEGACY `generate_once` forward (test
    /// h77), which differs from the slot-aware forward by a pre-existing
    /// numeric delta (under separate investigation per §0.12). Special-
    /// casing N=1 to legacy to force SerialFifo-equivalence was REJECTED
    /// (ruling (a)) — it would hide that the batched path runs entirely on
    /// the slot-aware forward.
    ///
    /// Falsifies any wrapper the scheduler-driven loop introduces that
    /// mutates the N=1 observable result (dropped token, decode-bound
    /// off-by-one, sampler-state shift, reasoning-count drift).
    #[test]
    fn slot_aware_n1_byte_equivalent_to_serial_slot_aware() {
        if byte_equiv_skip_unless_gated("slot_aware_n1_byte_equivalent_to_serial_slot_aware") {
            return;
        }
        let gguf_path: PathBuf = std::env::var(BYTE_EQUIV_E2E_GGUF_ENV)
            .map(PathBuf::from)
            .expect("HF2Q_BYTE_EQUIV_E2E_GGUF set");
        assert!(gguf_path.exists(), "GGUF missing: {}", gguf_path.display());

        let load_opts = LoadOptions {
            model_path: gguf_path.clone(),
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };
        let prompt_tokens: Vec<u32> = vec![1u32, 2, 3, 4, 5];
        let params = SamplingParams {
            temperature: 0.0,
            max_tokens: 16,
            ..Default::default()
        };

        // Reference: the serial slot-aware fn inline (same forward path).
        let r_ref = gemma4_serial_slot_aware_ref(&load_opts, &prompt_tokens, &params);

        // F1 loop at N=1.
        let loaded_slot = LoadedModel::load(&load_opts).expect("load slot");
        let engine_slot =
            Engine::spawn_with_mode(loaded_slot, 4, None, EngineMode::SlotAware { max_slots: 1 })
                .expect("spawn SlotAware{max_slots:1}");

        let rt = tokio::runtime::Builder::new_current_thread()
            .enable_all()
            .build()
            .expect("rt");
        let r_slot = rt
            .block_on(engine_slot.generate(prompt_tokens, params))
            .expect("slot generate");

        assert!(
            !r_ref.text.is_empty() || r_ref.completion_tokens > 0,
            "vacuous: ref produced no output"
        );
        assert_genresult_byte_equal(
            &r_slot,
            &r_ref,
            "ADR-040 F1 AC4 (b) — SlotAware N=1 loop vs serial slot-aware ref",
        );

        rt.block_on(engine_slot.shutdown()).expect("shutdown slot");
    }

    /// DIAGNOSTIC (ADR-040 F1 blocker triage) — is my F1 loop a FAITHFUL
    /// driver of the slot-aware forward? Compares SlotAware{1} (my loop)
    /// against the EXISTING serial reference `generate_gemma4_once_slot_aware`
    /// (same forward path, run inline). If these match, the N=1-vs-SerialFifo
    /// divergence is purely the legacy-vs-slot-aware forward delta (h77),
    /// not a bug in my loop. Gemma 4 only.
    #[test]
    fn slot_aware_n1_matches_serial_slot_aware_ref() {
        if byte_equiv_skip_unless_gated("slot_aware_n1_matches_serial_slot_aware_ref") {
            return;
        }
        let gguf_path: PathBuf = std::env::var(BYTE_EQUIV_E2E_GGUF_ENV)
            .map(PathBuf::from)
            .expect("HF2Q_BYTE_EQUIV_E2E_GGUF set");
        assert!(gguf_path.exists(), "GGUF missing: {}", gguf_path.display());
        let load_opts = LoadOptions {
            model_path: gguf_path.clone(),
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };
        let prompt_tokens: Vec<u32> = vec![1u32, 2, 3, 4, 5];
        let params = SamplingParams {
            temperature: 0.0,
            max_tokens: 16,
            ..Default::default()
        };

        // (1) My F1 loop via SlotAware{1}.
        let loaded_slot = LoadedModel::load(&load_opts).expect("load slot");
        let engine_slot =
            Engine::spawn_with_mode(loaded_slot, 4, None, EngineMode::SlotAware { max_slots: 1 })
                .expect("spawn SlotAware{1}");
        let rt = tokio::runtime::Builder::new_current_thread()
            .enable_all()
            .build()
            .expect("rt");
        let r_loop = rt
            .block_on(engine_slot.generate(prompt_tokens.clone(), params.clone()))
            .expect("loop generate");
        rt.block_on(engine_slot.shutdown()).expect("shutdown slot");

        // (2) The existing serial slot-aware ref, run inline at SlotId(0) on
        // a freshly-provisioned multi-seq KV.
        let mut loaded_ref = LoadedModel::load(&load_opts).expect("load ref");
        let LoadedModel::Gemma(g) = &mut loaded_ref else {
            panic!("expected Gemma GGUF")
        };
        // Provision the slot-aware multi-seq KV the way SlotAware spawn does.
        g.provision_multi_seq_kv_for_slot_aware(1)
            .expect("provision multi-seq KV n_seqs=1");
        let mut kv = g.multi_seq_kv.take().expect("multi_seq_kv provisioned");
        let mut hybrid = g.multi_seq_kv_hybrid.take();
        let mut dense = g.multi_seq_kv_dense.take();
        let mut mlx = g.multi_seq_kv_mlx.take();
        let r_ref = generate_gemma4_once_slot_aware(
            g,
            &prompt_tokens,
            &params,
            None,
            &mut kv,
            hybrid.as_mut(),
            dense.as_mut(),
            mlx.as_mut(),
            SlotId(0),
        )
        .expect("serial slot-aware ref");
        g.multi_seq_kv = Some(kv);
        g.multi_seq_kv_hybrid = hybrid;
        g.multi_seq_kv_dense = dense;
        g.multi_seq_kv_mlx = mlx;

        assert_genresult_byte_equal(
            &r_loop,
            &r_ref,
            "ADR-040 F1 DIAGNOSTIC — SlotAware{1} loop vs serial generate_gemma4_once_slot_aware",
        );
    }

    // =====================================================================
    // ADR-040 Phase F M1 BLOCKING INVESTIGATION — logit-level characterization
    // of the LEGACY (`generate_once` → forward_prefill_batched +
    // forward_decode) vs SLOT-AWARE (`forward_prefill_with_soft_tokens_
    // slot_aware` + forward_decode_slot_aware) greedy forward delta that h77
    // pinned but never root-caused.
    //
    // This test runs the SAME greedy prompt through BOTH forward paths,
    // capturing the raw `logits_view()` vector at EVERY decode position, then
    // quantifies:
    //   • max abs logit diff per position (and global max),
    //   • mean abs logit diff per position,
    //   • the FIRST decode position where the greedy argmax flips,
    //   • for any flip: the top-2 logit gap on BOTH paths at that position
    //     (near-tie ⇒ benign quant-noise flip; confident ⇒ structural bug).
    //
    // Both paths read the SAME `loaded.weights.activations.logits` buffer via
    // `logits_view()`, so we run them on TWO independent model instances and
    // snapshot the logits into owned Vecs immediately after each forward call
    // (before the next call overwrites the buffer). Greedy only (T=0) so the
    // token stream is deterministic on each path and the per-position compare
    // is apples-to-apples (we feed each path ITS OWN argmax forward, and also
    // record a "teacher-forced" compare driving BOTH paths with the LEGACY
    // token stream so a position-N logit delta is not confounded by a
    // position<N token divergence).
    //
    // Gated identically to the sibling E2E tests: HF2Q_BYTE_EQUIV_E2E=1 +
    // HF2Q_BYTE_EQUIV_E2E_GGUF=<path>. Run under the production-default
    // regime (HF2Q_TQ_CODEBOOK_BITS=8, HF2Q_HYBRID_KV unset=default-on).
    //
    // This is a DIAGNOSTIC test: it does not assert a tight bound (the whole
    // point is to MEASURE the delta). It asserts only sanity invariants
    // (non-empty logits, equal vocab) and PRINTS the full quantitative table
    // to stderr for the ADR note. A loose upper-bound assert guards against a
    // catastrophic regression (max logit delta > 5.0 would indicate a real
    // bug, not quant noise).
    // =====================================================================

    /// Snapshot helper: argmax + top-2 gap of a logits slice.
    fn argmax_and_top2_gap(logits: &[f32]) -> (u32, f32, f32) {
        // Returns (argmax_id, max_logit, gap_to_second).
        let mut best_i = 0usize;
        let mut best_v = f32::NEG_INFINITY;
        let mut second_v = f32::NEG_INFINITY;
        for (i, &v) in logits.iter().enumerate() {
            if v > best_v {
                second_v = best_v;
                best_v = v;
                best_i = i;
            } else if v > second_v {
                second_v = v;
            }
        }
        (best_i as u32, best_v, best_v - second_v)
    }

    /// Capture per-position logits for the LEGACY path (mirrors
    /// `generate_once` greedy fast-path: forward_prefill_batched then
    /// forward_decode loop). Drives the path with `driver_tokens` if Some
    /// (teacher-forced), else with its own greedy argmax. Returns
    /// (greedy_token_stream, per_position_logits) where position 0 is the
    /// prefill output (logits over the last prompt token) and position k is
    /// the logits AFTER feeding generated token k-1 via forward_decode.
    fn capture_legacy_logits(
        g: &mut GemmaLoadedModel,
        prompt_tokens: &[u32],
        max_tokens: usize,
        driver_tokens: Option<&[u32]>,
    ) -> (Vec<u32>, Vec<Vec<f32>>) {
        let mut logits_per_pos: Vec<Vec<f32>> = Vec::with_capacity(max_tokens);
        let mut greedy_stream: Vec<u32> = Vec::with_capacity(max_tokens);

        // Prefill (mirror generate_once default: forward_prefill_batched).
        let prefill_argmax = g
            .weights
            .forward_prefill_batched(prompt_tokens, max_tokens, 0, &mut g.ctx)
            .expect("legacy forward_prefill_batched");
        let l0 = g
            .weights
            .logits_view()
            .expect("legacy prefill logits")
            .to_vec();
        let (am0, _, _) = argmax_and_top2_gap(&l0);
        assert_eq!(
            am0, prefill_argmax,
            "legacy: logits argmax != kernel prefill argmax"
        );
        logits_per_pos.push(l0);
        greedy_stream.push(prefill_argmax);

        // Token fed at decode step i is driver_tokens[i] if teacher-forced,
        // else our own greedy stream.
        let mut next_token = driver_tokens.map(|d| d[0]).unwrap_or(prefill_argmax);
        for step in 1..max_tokens {
            let pos = prompt_tokens.len() + step - 1;
            let mut p: Option<crate::inference::models::gemma4::profile::TokenProfile> = None;
            let greedy = g
                .weights
                .forward_decode(next_token, pos, &mut g.ctx, &mut p)
                .expect("legacy forward_decode");
            let lk = g
                .weights
                .logits_view()
                .expect("legacy decode logits")
                .to_vec();
            let (amk, _, _) = argmax_and_top2_gap(&lk);
            assert_eq!(amk, greedy, "legacy: decode logits argmax != kernel greedy");
            logits_per_pos.push(lk);
            greedy_stream.push(greedy);
            next_token = match driver_tokens {
                Some(d) => d[step],
                None => greedy,
            };
        }
        (greedy_stream, logits_per_pos)
    }

    /// Capture per-position logits for the SLOT-AWARE path (mirrors
    /// `generate_gemma4_once_slot_aware` greedy fast-path:
    /// forward_prefill_with_soft_tokens_slot_aware then
    /// forward_decode_slot_aware loop) at SlotId(0). Same driver semantics
    /// as `capture_legacy_logits`.
    #[allow(clippy::too_many_arguments)]
    fn capture_slot_aware_logits(
        g: &mut GemmaLoadedModel,
        prompt_tokens: &[u32],
        max_tokens: usize,
        driver_tokens: Option<&[u32]>,
        kv: &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqHbKvBuffers>,
        mut hybrid: Option<
            &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqHybridKvBuffers>,
        >,
        mut dense: Option<
            &mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqDenseKvBuffers>,
        >,
        mut mlx: Option<&mut Vec<crate::inference::models::gemma4::kv_cache::MultiSeqMlxKvCache>>,
    ) -> (Vec<u32>, Vec<Vec<f32>>) {
        let slot = SlotId(0);
        // Entry reset on every per-layer buffer (mirror the orchestrator).
        for buf in kv.iter_mut() {
            buf.reset_for_slot(slot).expect("slot-aware entry reset hb");
        }
        if let Some(ref mut h) = hybrid {
            for buf in h.iter_mut() {
                buf.reset_for_slot(slot)
                    .expect("slot-aware entry reset hybrid");
            }
        }

        let mut logits_per_pos: Vec<Vec<f32>> = Vec::with_capacity(max_tokens);
        let mut greedy_stream: Vec<u32> = Vec::with_capacity(max_tokens);

        let prefill_argmax = g
            .weights
            .forward_prefill_with_soft_tokens_slot_aware(
                prompt_tokens,
                &[],
                max_tokens,
                &mut g.ctx,
                slot,
                kv,
                hybrid.as_deref_mut(),
                dense.as_deref_mut(),
                mlx.as_deref_mut(),
            )
            .expect("slot-aware prefill");
        let l0 = g
            .weights
            .logits_view()
            .expect("slot-aware prefill logits")
            .to_vec();
        let (am0, _, _) = argmax_and_top2_gap(&l0);
        assert_eq!(
            am0, prefill_argmax,
            "slot-aware: logits argmax != kernel prefill argmax"
        );
        logits_per_pos.push(l0);
        greedy_stream.push(prefill_argmax);

        let mut next_token = driver_tokens.map(|d| d[0]).unwrap_or(prefill_argmax);
        for step in 1..max_tokens {
            let pos = prompt_tokens.len() + step - 1;
            let mut p: Option<crate::inference::models::gemma4::profile::TokenProfile> = None;
            let greedy = g
                .weights
                .forward_decode_slot_aware(
                    next_token,
                    pos,
                    &mut g.ctx,
                    &mut p,
                    slot,
                    kv,
                    hybrid.as_deref_mut(),
                    dense.as_deref_mut(),
                    mlx.as_deref_mut(),
                )
                .expect("slot-aware forward_decode");
            let lk = g
                .weights
                .logits_view()
                .expect("slot-aware decode logits")
                .to_vec();
            let (amk, _, _) = argmax_and_top2_gap(&lk);
            assert_eq!(
                amk, greedy,
                "slot-aware: decode logits argmax != kernel greedy"
            );
            logits_per_pos.push(lk);
            greedy_stream.push(greedy);
            next_token = match driver_tokens {
                Some(d) => d[step],
                None => greedy,
            };
        }
        (greedy_stream, logits_per_pos)
    }

    #[test]
    fn adr040_f_m1_legacy_vs_slot_aware_logit_characterization() {
        if byte_equiv_skip_unless_gated("adr040_f_m1_legacy_vs_slot_aware_logit_characterization") {
            return;
        }
        let gguf_path: PathBuf = std::env::var(BYTE_EQUIV_E2E_GGUF_ENV)
            .map(PathBuf::from)
            .expect("HF2Q_BYTE_EQUIV_E2E_GGUF set");
        assert!(gguf_path.exists(), "GGUF missing: {}", gguf_path.display());
        let load_opts = LoadOptions {
            model_path: gguf_path.clone(),
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };
        let prompt_tokens: Vec<u32> = vec![1u32, 2, 3, 4, 5];
        let max_tokens = 16usize;

        eprintln!("\n==== ADR-040 F M1 logit characterization ====");
        eprintln!(
            "regime: HF2Q_TQ_CODEBOOK_BITS={:?} HF2Q_HYBRID_KV={:?} HF2Q_USE_DENSE={:?}",
            std::env::var("HF2Q_TQ_CODEBOOK_BITS").ok(),
            std::env::var("HF2Q_HYBRID_KV").ok(),
            std::env::var("HF2Q_USE_DENSE").ok(),
        );

        // ── PASS 1: each path drives its OWN greedy argmax (real generation) ──
        let (legacy_stream, legacy_logits) = {
            let mut loaded = LoadedModel::load(&load_opts).expect("load legacy");
            let LoadedModel::Gemma(g) = &mut loaded else {
                panic!("expected Gemma GGUF")
            };
            capture_legacy_logits(g, &prompt_tokens, max_tokens, None)
        };
        let (slot_stream, slot_logits) = {
            let mut loaded = LoadedModel::load(&load_opts).expect("load slot");
            let LoadedModel::Gemma(g) = &mut loaded else {
                panic!("expected Gemma GGUF")
            };
            g.provision_multi_seq_kv_for_slot_aware(1)
                .expect("provision multi-seq KV");
            let mut kv = g.multi_seq_kv.take().expect("kv");
            let mut hybrid = g.multi_seq_kv_hybrid.take();
            let mut dense = g.multi_seq_kv_dense.take();
            let mut mlx = g.multi_seq_kv_mlx.take();
            let r = capture_slot_aware_logits(
                g,
                &prompt_tokens,
                max_tokens,
                None,
                &mut kv,
                hybrid.as_mut(),
                dense.as_mut(),
                mlx.as_mut(),
            );
            g.multi_seq_kv = Some(kv);
            g.multi_seq_kv_hybrid = hybrid;
            g.multi_seq_kv_dense = dense;
            g.multi_seq_kv_mlx = mlx;
            r
        };

        let vocab = legacy_logits[0].len();
        assert_eq!(
            vocab,
            slot_logits[0].len(),
            "vocab size mismatch between paths"
        );
        assert!(vocab > 0, "empty logits");
        assert_eq!(legacy_logits.len(), max_tokens);
        assert_eq!(slot_logits.len(), max_tokens);

        eprintln!("\n-- PASS 1: self-driven greedy streams --");
        eprintln!("legacy stream:    {:?}", legacy_stream);
        eprintln!("slot-aware stream:{:?}", slot_stream);
        let mut first_stream_div: Option<usize> = None;
        for i in 0..max_tokens {
            if legacy_stream[i] != slot_stream[i] {
                first_stream_div = Some(i);
                break;
            }
        }
        eprintln!(
            "first greedy-stream divergence position: {:?}",
            first_stream_div
        );

        // ── PASS 2: TEACHER-FORCED on the legacy token stream so a
        // position-N logit delta is NOT confounded by an earlier token
        // divergence feeding a different KV history into the two paths.
        // Both paths consume `legacy_stream` as input; we then compare
        // logits position-by-position over identical input histories. THIS
        // is the load-bearing measurement for root-cause. ──
        let (_, legacy_tf) = {
            let mut loaded = LoadedModel::load(&load_opts).expect("load legacy tf");
            let LoadedModel::Gemma(g) = &mut loaded else {
                panic!("expected Gemma GGUF")
            };
            capture_legacy_logits(g, &prompt_tokens, max_tokens, Some(&legacy_stream))
        };
        let (_, slot_tf) = {
            let mut loaded = LoadedModel::load(&load_opts).expect("load slot tf");
            let LoadedModel::Gemma(g) = &mut loaded else {
                panic!("expected Gemma GGUF")
            };
            g.provision_multi_seq_kv_for_slot_aware(1)
                .expect("provision multi-seq KV tf");
            let mut kv = g.multi_seq_kv.take().expect("kv tf");
            let mut hybrid = g.multi_seq_kv_hybrid.take();
            let mut dense = g.multi_seq_kv_dense.take();
            let mut mlx = g.multi_seq_kv_mlx.take();
            let r = capture_slot_aware_logits(
                g,
                &prompt_tokens,
                max_tokens,
                Some(&legacy_stream),
                &mut kv,
                hybrid.as_mut(),
                dense.as_mut(),
                mlx.as_mut(),
            );
            g.multi_seq_kv = Some(kv);
            g.multi_seq_kv_hybrid = hybrid;
            g.multi_seq_kv_dense = dense;
            g.multi_seq_kv_mlx = mlx;
            r
        };

        eprintln!(
            "\n-- PASS 2: TEACHER-FORCED (both paths fed legacy stream) --\n\
             pos | max_abs_diff | mean_abs_diff | L_argmax(gap) | S_argmax(gap) | flip?"
        );
        let mut global_max_diff = 0.0f32;
        let mut global_max_diff_pos = 0usize;
        let mut first_argmax_flip: Option<usize> = None;
        let mut flip_details: Vec<(usize, f32, f32)> = Vec::new();
        let mut sum_mean_over_pos = 0.0f64;
        for pos in 0..max_tokens {
            let a = &legacy_tf[pos];
            let b = &slot_tf[pos];
            let mut max_abs = 0.0f32;
            let mut sum_abs = 0.0f64;
            for j in 0..vocab {
                let d = (a[j] - b[j]).abs();
                if d > max_abs {
                    max_abs = d;
                }
                sum_abs += d as f64;
            }
            let mean_abs = (sum_abs / vocab as f64) as f32;
            sum_mean_over_pos += mean_abs as f64;
            if max_abs > global_max_diff {
                global_max_diff = max_abs;
                global_max_diff_pos = pos;
            }
            let (la, _lv, lgap) = argmax_and_top2_gap(a);
            let (sa, _sv, sgap) = argmax_and_top2_gap(b);
            let flip = la != sa;
            if flip && first_argmax_flip.is_none() {
                first_argmax_flip = Some(pos);
            }
            if flip {
                flip_details.push((pos, lgap, sgap));
            }
            eprintln!(
                "{:3} | {:12.6} | {:13.8} | {:6}({:8.5}) | {:6}({:8.5}) | {}",
                pos,
                max_abs,
                mean_abs,
                la,
                lgap,
                sa,
                sgap,
                if flip { "FLIP" } else { "" }
            );
        }
        let mean_mean = sum_mean_over_pos / max_tokens as f64;
        eprintln!(
            "\nGLOBAL: max_abs_logit_diff={:.6} @pos {} | avg(mean_abs_diff/pos)={:.8}",
            global_max_diff, global_max_diff_pos, mean_mean
        );
        eprintln!("first teacher-forced argmax flip: {:?}", first_argmax_flip);
        eprintln!(
            "all flips (pos, legacy_top2_gap, slot_top2_gap): {:?}",
            flip_details
        );

        // Characterize: are flips on near-ties (benign) or confident (bug)?
        for (pos, lgap, sgap) in &flip_details {
            let near_tie = *lgap < global_max_diff.max(1e-3) || *sgap < global_max_diff.max(1e-3);
            eprintln!(
                "  flip @pos {}: legacy_gap={:.6} slot_gap={:.6} -> {}",
                pos,
                lgap,
                sgap,
                if near_tie {
                    "NEAR-TIE (gap <= logit-noise => benign finite-precision flip)"
                } else {
                    "CONFIDENT (gap >> logit-noise => STRUCTURAL — investigate!)"
                }
            );
        }
        if global_max_diff >= 5.0 {
            eprintln!(
                "*** VERDICT SIGNAL: max abs logit diff {:.4} >= 5.0 at PREFILL/early \
                 positions — this is NOT plausible 8-bit-codebook V-quant rounding \
                 noise. The slot-aware forward is computing a STRUCTURALLY different \
                 result, not merely a finite-precision variant. See the \
                 adr040_f_m1_prefill_disambiguation companion test for the \
                 batched-vs-nonbatched-vs-slot isolation. ***",
                global_max_diff
            );
        }
        eprintln!("==== end ADR-040 F M1 characterization ====\n");

        // Sanity invariant: logits must be finite + non-degenerate. We do
        // NOT assert a tight bound here — this is a measurement test whose
        // job is to MEASURE the delta. The companion disambiguation test
        // pins the root-cause axis.
        assert!(
            global_max_diff.is_finite(),
            "ADR-040 F M1: non-finite logit delta"
        );
    }

    /// ADR-040 Phase F M1 — PREFILL-ONLY disambiguation. The
    /// characterization test above shows a ~20-logit delta at decode
    /// position 0 (the PREFILL output, before ANY KV-quantized decode
    /// round-trip), which rules out TQ-V-quant decode noise as the cause.
    /// This test isolates the prefill delta across the THREE prefill kernels
    /// the two generate paths actually use:
    ///   (A) legacy DEFAULT: `forward_prefill_batched`            (generate_once)
    ///   (B) legacy non-batched: `forward_prefill_with_soft_tokens_resume`
    ///   (C) slot-aware: `forward_prefill_with_soft_tokens_slot_aware`
    ///       (which internally DELEGATES to (B) after mounting per-slot KV
    ///        views — see forward_prefill.rs:3676)
    ///
    /// Greedy prefill argmax + full logits captured for each. We then report
    /// max abs logit diff for A-vs-B (batched-vs-nonbatched axis) and
    /// B-vs-C (slot-view-mount axis). This pinpoints whether the delta lives
    /// in the batched/non-batched prefill split (a pre-existing axis,
    /// orthogonal to slot-awareness) or in the slot-aware KV-view mount
    /// itself.
    #[test]
    fn adr040_f_m1_prefill_disambiguation() {
        if byte_equiv_skip_unless_gated("adr040_f_m1_prefill_disambiguation") {
            return;
        }
        let gguf_path: PathBuf = std::env::var(BYTE_EQUIV_E2E_GGUF_ENV)
            .map(PathBuf::from)
            .expect("HF2Q_BYTE_EQUIV_E2E_GGUF set");
        assert!(gguf_path.exists(), "GGUF missing: {}", gguf_path.display());
        let load_opts = LoadOptions {
            model_path: gguf_path.clone(),
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };
        let prompt: Vec<u32> = vec![1u32, 2, 3, 4, 5];
        let max_tokens = 16usize;

        // (A) legacy batched prefill.
        let (a_argmax, a_logits) = {
            let mut loaded = LoadedModel::load(&load_opts).expect("load A");
            let LoadedModel::Gemma(g) = &mut loaded else {
                panic!("Gemma")
            };
            let am = g
                .weights
                .forward_prefill_batched(&prompt, max_tokens, 0, &mut g.ctx)
                .expect("A prefill_batched");
            (am, g.weights.logits_view().expect("A logits").to_vec())
        };
        // (B) legacy non-batched resume prefill.
        let (b_argmax, b_logits) = {
            let mut loaded = LoadedModel::load(&load_opts).expect("load B");
            let LoadedModel::Gemma(g) = &mut loaded else {
                panic!("Gemma")
            };
            let am = g
                .weights
                .forward_prefill_with_soft_tokens_resume(
                    &prompt,
                    &[],
                    max_tokens,
                    &mut g.ctx,
                    None,
                    false,
                )
                .expect("B prefill_resume");
            (am, g.weights.logits_view().expect("B logits").to_vec())
        };
        // (C) slot-aware prefill.
        let (c_argmax, c_logits) = {
            let mut loaded = LoadedModel::load(&load_opts).expect("load C");
            let LoadedModel::Gemma(g) = &mut loaded else {
                panic!("Gemma")
            };
            g.provision_multi_seq_kv_for_slot_aware(1)
                .expect("provision");
            let mut kv = g.multi_seq_kv.take().expect("kv");
            let mut hybrid = g.multi_seq_kv_hybrid.take();
            let mut dense = g.multi_seq_kv_dense.take();
            let mut mlx = g.multi_seq_kv_mlx.take();
            for buf in kv.iter_mut() {
                buf.reset_for_slot(SlotId(0)).expect("reset hb");
            }
            if let Some(ref mut h) = hybrid {
                for buf in h.iter_mut() {
                    buf.reset_for_slot(SlotId(0)).expect("reset hybrid");
                }
            }
            let am = g
                .weights
                .forward_prefill_with_soft_tokens_slot_aware(
                    &prompt,
                    &[],
                    max_tokens,
                    &mut g.ctx,
                    SlotId(0),
                    &mut kv,
                    hybrid.as_mut(),
                    dense.as_mut(),
                    mlx.as_mut(),
                )
                .expect("C prefill_slot_aware");
            let lg = g.weights.logits_view().expect("C logits").to_vec();
            g.multi_seq_kv = Some(kv);
            g.multi_seq_kv_hybrid = hybrid;
            g.multi_seq_kv_dense = dense;
            g.multi_seq_kv_mlx = mlx;
            (am, lg)
        };

        let vocab = a_logits.len();
        assert_eq!(vocab, b_logits.len());
        assert_eq!(vocab, c_logits.len());
        let max_abs = |x: &[f32], y: &[f32]| -> f32 {
            x.iter()
                .zip(y.iter())
                .map(|(p, q)| (p - q).abs())
                .fold(0.0f32, f32::max)
        };
        let ab = max_abs(&a_logits, &b_logits);
        let bc = max_abs(&b_logits, &c_logits);
        let ac = max_abs(&a_logits, &c_logits);
        eprintln!("\n==== ADR-040 F M1 PREFILL disambiguation ====");
        eprintln!("prompt={:?} vocab={}", prompt, vocab);
        eprintln!("(A) legacy batched      prefill argmax = {}", a_argmax);
        eprintln!("(B) legacy non-batched  prefill argmax = {}", b_argmax);
        eprintln!("(C) slot-aware          prefill argmax = {}", c_argmax);
        eprintln!("max|A-B| (batched vs non-batched)        = {:.6}", ab);
        eprintln!("max|B-C| (non-batched vs slot-aware MOUNT)= {:.6}", bc);
        eprintln!("max|A-C| (legacy-default vs slot-aware)   = {:.6}", ac);
        eprintln!(
            "INTERPRETATION: if max|B-C| ~ 0 then slot-aware prefill == legacy \
             non-batched prefill (delta lives in the batched/non-batched axis, \
             which is ORTHOGONAL to slot-awareness). If max|B-C| is large, the \
             slot-view KV mount itself perturbs the prefill."
        );
        eprintln!("==== end PREFILL disambiguation ====\n");
        assert!(ab.is_finite() && bc.is_finite() && ac.is_finite());

        // ── FULL-DECODE B-vs-C: confirm slot-aware tracks legacy NON-BATCHED
        // through the entire greedy decode (not just prefill). Teacher-force
        // BOTH on the legacy NON-BATCHED greedy stream so the input history
        // is identical, then compare logits at every position. If max|B-C|
        // stays ~0 across decode, the slot-aware forward is byte-faithful to
        // the legacy non-batched forward — the only delta vs the SerialFifo
        // DEFAULT is the batched-vs-non-batched prefill axis pinned above. ──
        let (b_stream, _) = {
            let mut loaded = LoadedModel::load(&load_opts).expect("load Bstream");
            let LoadedModel::Gemma(g) = &mut loaded else {
                panic!("Gemma")
            };
            // Drive legacy NON-BATCHED greedy: prefill_resume then forward_decode.
            let mut stream = Vec::with_capacity(max_tokens);
            let am = g
                .weights
                .forward_prefill_with_soft_tokens_resume(
                    &prompt,
                    &[],
                    max_tokens,
                    &mut g.ctx,
                    None,
                    false,
                )
                .expect("Bstream prefill");
            stream.push(am);
            let mut nt = am;
            for step in 1..max_tokens {
                let pos = prompt.len() + step - 1;
                let mut p: Option<crate::inference::models::gemma4::profile::TokenProfile> = None;
                let g_tok = g
                    .weights
                    .forward_decode(nt, pos, &mut g.ctx, &mut p)
                    .expect("Bstream decode");
                stream.push(g_tok);
                nt = g_tok;
            }
            (stream, ())
        };
        // B-path teacher-forced logits.
        let b_tf = {
            let mut loaded = LoadedModel::load(&load_opts).expect("load Btf");
            let LoadedModel::Gemma(g) = &mut loaded else {
                panic!("Gemma")
            };
            capture_legacy_logits_nonbatched(g, &prompt, max_tokens, &b_stream)
        };
        // C-path (slot-aware) teacher-forced logits.
        let c_tf = {
            let mut loaded = LoadedModel::load(&load_opts).expect("load Ctf");
            let LoadedModel::Gemma(g) = &mut loaded else {
                panic!("Gemma")
            };
            g.provision_multi_seq_kv_for_slot_aware(1)
                .expect("provision Ctf");
            let mut kv = g.multi_seq_kv.take().expect("kv");
            let mut hybrid = g.multi_seq_kv_hybrid.take();
            let mut dense = g.multi_seq_kv_dense.take();
            let mut mlx = g.multi_seq_kv_mlx.take();
            let (_, lg) = capture_slot_aware_logits(
                g,
                &prompt,
                max_tokens,
                Some(&b_stream),
                &mut kv,
                hybrid.as_mut(),
                dense.as_mut(),
                mlx.as_mut(),
            );
            g.multi_seq_kv = Some(kv);
            g.multi_seq_kv_hybrid = hybrid;
            g.multi_seq_kv_dense = dense;
            g.multi_seq_kv_mlx = mlx;
            lg
        };
        let mut decode_max = 0.0f32;
        let mut decode_max_pos = 0usize;
        for pos in 0..max_tokens {
            let d = max_abs(&b_tf[pos], &c_tf[pos]);
            if d > decode_max {
                decode_max = d;
                decode_max_pos = pos;
            }
        }
        eprintln!("==== ADR-040 F M1 FULL-DECODE B(non-batched legacy) vs C(slot-aware) ====");
        eprintln!("legacy non-batched greedy stream: {:?}", b_stream);
        eprintln!(
            "max|B-C| over ALL {} decode positions (teacher-forced) = {:.8} @pos {}",
            max_tokens, decode_max, decode_max_pos
        );
        eprintln!(
            "INTERPRETATION: ~0 ⇒ slot-aware forward is byte-faithful to the legacy \
             NON-BATCHED forward end-to-end; the h77 SerialFifo delta is SOLELY the \
             batched-vs-non-batched prefill axis (orthogonal to slot-awareness). The \
             TQ-HB-V 8-bit KV quant introduces NO observable decode delta vs the \
             legacy hybrid KV (both use the SAME hybrid F16-K + TQ-HB-V cache)."
        );
        eprintln!("==== end FULL-DECODE B-vs-C ====\n");
        assert!(decode_max.is_finite());
    }

    /// Helper: capture legacy NON-BATCHED prefill+decode logits, teacher-forced.
    fn capture_legacy_logits_nonbatched(
        g: &mut GemmaLoadedModel,
        prompt_tokens: &[u32],
        max_tokens: usize,
        driver_tokens: &[u32],
    ) -> Vec<Vec<f32>> {
        let mut out: Vec<Vec<f32>> = Vec::with_capacity(max_tokens);
        let _am = g
            .weights
            .forward_prefill_with_soft_tokens_resume(
                prompt_tokens,
                &[],
                max_tokens,
                &mut g.ctx,
                None,
                false,
            )
            .expect("nb prefill");
        out.push(g.weights.logits_view().expect("nb prefill logits").to_vec());
        let mut nt = driver_tokens[0];
        for step in 1..max_tokens {
            let pos = prompt_tokens.len() + step - 1;
            let mut p: Option<crate::inference::models::gemma4::profile::TokenProfile> = None;
            let _g = g
                .weights
                .forward_decode(nt, pos, &mut g.ctx, &mut p)
                .expect("nb decode");
            out.push(g.weights.logits_view().expect("nb decode logits").to_vec());
            nt = driver_tokens[step];
        }
        out
    }

    /// F1 AC1+AC2 — N concurrent distinct prompts through SlotAware each
    /// match their own SerialFifo reference (per-slot independence + no
    /// cross-slot leakage). Drives 4 distinct prompts concurrently on a
    /// multi-thread runtime through one `SlotAware { max_slots: 4 }` engine,
    /// then computes the serial reference for each prompt and asserts
    /// per-prompt byte equality. A `assert_ne` guard rejects a fixture
    /// where the distinct prompts collapse to identical output (which would
    /// make the cross-slot isolation assertions vacuous).
    #[test]
    fn slot_aware_n4_per_slot_parity_vs_serial() {
        if byte_equiv_skip_unless_gated("slot_aware_n4_per_slot_parity_vs_serial") {
            return;
        }
        let gguf_path: PathBuf = std::env::var(BYTE_EQUIV_E2E_GGUF_ENV)
            .map(PathBuf::from)
            .expect("HF2Q_BYTE_EQUIV_E2E_GGUF set");
        assert!(gguf_path.exists(), "GGUF missing: {}", gguf_path.display());
        let load_opts = LoadOptions {
            model_path: gguf_path.clone(),
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };

        // Four distinct greedy prompts.
        let prompts: Vec<Vec<u32>> = vec![
            vec![1, 2, 3],
            vec![4, 5, 6, 7],
            vec![8, 9],
            vec![10, 11, 12, 13, 14],
        ];
        let params = SamplingParams {
            temperature: 0.0,
            max_tokens: 24,
            ..Default::default()
        };

        // Serial slot-aware references (SAME forward path as the SlotAware
        // loop) — the AC4-correct bar, decoupled from the legacy-forward
        // delta (h77). One independent model per prompt.
        let serial_refs: Vec<GenerationResult> = prompts
            .iter()
            .map(|p| gemma4_serial_slot_aware_ref(&load_opts, p, &params))
            .collect();
        let rt = tokio::runtime::Builder::new_multi_thread()
            .worker_threads(4)
            .enable_all()
            .build()
            .expect("rt");

        // Vacuous-test guard: distinct prompts must not collapse to the
        // same output, else cross-slot isolation is untested.
        assert_ne!(
            serial_refs[0].text, serial_refs[1].text,
            "vacuous: prompts 0 and 1 produced identical serial output"
        );

        // Concurrent SlotAware run: clone the engine handle per prompt and
        // drive all generates concurrently on the multi-thread runtime.
        let loaded_slot = LoadedModel::load(&load_opts).expect("load slot");
        let engine_slot =
            Engine::spawn_with_mode(loaded_slot, 8, None, EngineMode::SlotAware { max_slots: 4 })
                .expect("spawn SlotAware{max_slots:4}");
        let slot_results: Vec<GenerationResult> = rt.block_on(async {
            let mut handles = Vec::new();
            for p in prompts.iter().cloned() {
                let eng = engine_slot.clone();
                let pr = params.clone();
                handles.push(tokio::spawn(async move {
                    eng.generate(p, pr).await.expect("slot generate")
                }));
            }
            let mut out = Vec::new();
            for h in handles {
                out.push(h.await.expect("join"));
            }
            out
        });
        for (i, (slot, serial)) in slot_results.iter().zip(serial_refs.iter()).enumerate() {
            assert_genresult_byte_equal(
                slot,
                serial,
                &format!("ADR-040 F1 AC1/AC2 — SlotAware N=4 slot {i} vs its serial ref"),
            );
        }
        rt.block_on(engine_slot.shutdown()).expect("shutdown slot");
    }

    /// ADR-040 §0.16 `iter-F-batched-determinism-residual` FALSIFIER (2026-06-25).
    /// Bisects the trigger of the batched-body staggered flake. `n4_per_slot_parity`
    /// uses DISTINCT prompts (compares each slot only to its OWN serial ref → blind
    /// to slot-0-vs-slot-N divergence). This test uses an IDENTICAL prompt in all
    /// slots so any slot-index flavor split shows as a non-prefix of the single
    /// serial ref. Modes via env (batched body must be forced on, else no-op):
    ///   * default (MS=0, EVICT=0): simultaneous N=4, equal budget — **PASSES**.
    ///   * HF2Q_FALSIFIER_STAGGER_MS=40: staggered admission, equal budget — **PASSES** (3/3+).
    ///   * HF2Q_FALSIFIER_EVICT=1: slot 0 short→freed→5th reuses it, uniform long
    ///     budget for peers — **PASSES** (7/7).
    /// EXCLUSIONS PROVEN (2026-06-25): the residual is NOT same-prompt-alone, NOT
    /// staggered-admission-alone, NOT varying-N-by-join, NOT a single clean
    /// uniform-budget eviction. The live `staggered_eviction` test (which DOES fail
    /// ~13%) differs only by DISTINCT per-slot budgets [5/50/200/10] → eviction
    /// CHURN (multiple slots finishing+recycling at different ticks). That churn,
    /// not any single ingredient here, is the remaining trigger — consistent with
    /// codex H1 (stale batched-decode state inherited across the reset/reuse path).
    /// Kept as a regression guard for the same-prompt batched-body parity that
    /// `n4_per_slot_parity` cannot cover.
    #[test]
    fn slot_aware_n4_batched_body_same_prompt_parity_vs_serial() {
        if byte_equiv_skip_unless_gated("slot_aware_n4_batched_body_same_prompt_parity_vs_serial") {
            return;
        }
        if std::env::var("HF2Q_BATCHED_BODY").as_deref() != Ok("1") {
            eprintln!(
                "[skip] slot_aware_n4_batched_body_same_prompt_parity_vs_serial — \
                 set HF2Q_BATCHED_BODY=1 (+ HF2Q_BATCHED_ATTNPRE=1 HF2Q_BATCHED_FLASH=1) to run"
            );
            return;
        }
        let gguf_path: PathBuf = std::env::var(BYTE_EQUIV_E2E_GGUF_ENV)
            .map(PathBuf::from)
            .expect("HF2Q_BYTE_EQUIV_E2E_GGUF set");
        assert!(gguf_path.exists(), "GGUF missing: {}", gguf_path.display());
        let load_opts = LoadOptions {
            model_path: gguf_path.clone(),
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };

        // IDENTICAL prompt in all four slots. Serial ref at the LONGEST budget so
        // every slot's greedy output is a prefix of it.
        let prompt: Vec<u32> = vec![2, 4, 6, 8];
        let ref_params = SamplingParams {
            temperature: 0.0,
            max_tokens: 40,
            ..Default::default()
        };
        let serial = gemma4_serial_slot_aware_ref(&load_opts, &prompt, &ref_params);

        let rt = tokio::runtime::Builder::new_multi_thread()
            .worker_threads(4)
            .enable_all()
            .build()
            .expect("rt");
        let loaded_slot = LoadedModel::load(&load_opts).expect("load slot");
        let engine_slot =
            Engine::spawn_with_mode(loaded_slot, 8, None, EngineMode::SlotAware { max_slots: 4 })
                .expect("spawn SlotAware{max_slots:4}");
        // STAGGER admission: HF2Q_FALSIFIER_STAGGER_MS>0 inserts an increasing
        // pre-generate delay per slot so each prefills SEPARATELY (slot 0 solo,
        // then slot 1, …) — reproducing the eviction test's staggered prefill
        // WITHOUT eviction/varying-budget. If the simultaneous run (MS=0) passes
        // but the staggered run fails, the trigger is admission timing / prefill
        // ordering (codex H1: prior solo-prefill leaves stale batched-decode state).
        let stagger_ms: u64 = std::env::var("HF2Q_FALSIFIER_STAGGER_MS")
            .ok()
            .and_then(|s| s.parse().ok())
            .unwrap_or(0);
        // EVICTION mode: give slot 0 a SHORT budget so it finishes and frees its
        // slot mid-stream; a 5th same-prompt request is then admitted into the
        // recycled slot while the long slots keep decoding. This is the ONLY
        // differentiator left vs the (passing) simultaneous/staggered runs above.
        let evict = std::env::var("HF2Q_FALSIFIER_EVICT").as_deref() == Ok("1");
        let long = 40usize;
        let budgets: Vec<usize> = if evict {
            vec![4, long, long, long]
        } else {
            vec![24; 4]
        };
        let slot_results: Vec<GenerationResult> = rt.block_on(async {
            let mut handles = Vec::new();
            for (slot_i, &b) in budgets.iter().enumerate() {
                let eng = engine_slot.clone();
                let p = prompt.clone();
                let pr = SamplingParams {
                    temperature: 0.0,
                    max_tokens: b,
                    ..Default::default()
                };
                let si = slot_i as u64;
                handles.push(tokio::spawn(async move {
                    if stagger_ms > 0 {
                        tokio::time::sleep(std::time::Duration::from_millis(si * stagger_ms)).await;
                    }
                    eng.generate(p, pr).await.expect("slot generate")
                }));
            }
            // 5th request reuses the slot freed by the short budget-4 slot.
            if evict {
                let eng = engine_slot.clone();
                let p = prompt.clone();
                let pr = SamplingParams {
                    temperature: 0.0,
                    max_tokens: long,
                    ..Default::default()
                };
                handles.push(tokio::spawn(async move {
                    tokio::time::sleep(std::time::Duration::from_millis(60)).await;
                    eng.generate(p, pr).await.expect("slot5 generate")
                }));
            }
            let mut out = Vec::new();
            for h in handles {
                out.push(h.await.expect("join"));
            }
            out
        });
        // Greedy + same prompt ⇒ every slot's text MUST be a prefix of the serial
        // ref (each just stops at its own budget). Any divergence = the residual.
        for (i, slot) in slot_results.iter().enumerate() {
            assert!(
                serial.text.starts_with(&slot.text),
                "ADR-040 §0.16 — batched-body N=4 SAME-prompt (stagger_ms={stagger_ms} evict={evict}) \
                 slot {i} text is NOT a prefix of the serial ref (FAIL ⇒ this admission pattern is the trigger)\n  slot:   {:?}\n  serial: {:?}",
                slot.text, serial.text,
            );
        }
        rt.block_on(engine_slot.shutdown()).expect("shutdown slot");
    }

    /// ADR-040 Phase F `iter-F-n8parity` (2026-06-24, queen-led audit Worker B
    /// gap-closure) — the N=**8** analogue of `slot_aware_n4_per_slot_parity_vs_serial`.
    /// Phase F raised the live continuous-batching default to N=8
    /// (`ADR040_F_DEFAULT_CONTINUOUS_BATCHING_MAX_SLOTS`), but the parity bar
    /// above hard-codes `max_slots: 4`, leaving the shipped default's coherence
    /// *extrapolated* from the exact `MM_ROUTING_THRESHOLD(=8)` dispatch boundary
    /// rather than *proven*. This drives 8 distinct prompts concurrently through
    /// one `SlotAware { max_slots: 8 }` engine and asserts each slot is
    /// byte-identical to its independent serial slot-aware reference — closing
    /// the audit's "no N=8 byte-parity test" gap. Run through the batched body
    /// with `HF2Q_BATCHED_BODY=1` (+`_KVENC`/`_ATTNPRE`) to prove the S2/S3 path
    /// at the default width; gated by the shared E2E GGUF env like its N=4 peer.
    #[test]
    fn slot_aware_n8_per_slot_parity_vs_serial() {
        if byte_equiv_skip_unless_gated("slot_aware_n8_per_slot_parity_vs_serial") {
            return;
        }
        let gguf_path: PathBuf = std::env::var(BYTE_EQUIV_E2E_GGUF_ENV)
            .map(PathBuf::from)
            .expect("HF2Q_BYTE_EQUIV_E2E_GGUF set");
        assert!(gguf_path.exists(), "GGUF missing: {}", gguf_path.display());
        let load_opts = LoadOptions {
            model_path: gguf_path.clone(),
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };

        // Eight distinct greedy prompts (the N=4 set + four more distinct ones).
        let prompts: Vec<Vec<u32>> = vec![
            vec![1, 2, 3],
            vec![4, 5, 6, 7],
            vec![8, 9],
            vec![10, 11, 12, 13, 14],
            vec![15, 16],
            vec![17, 18, 19, 20],
            vec![21, 22, 23],
            vec![24, 25, 26, 27, 28],
        ];
        let params = SamplingParams {
            temperature: 0.0,
            max_tokens: 24,
            ..Default::default()
        };

        // Serial slot-aware references (AC4-correct bar; one model per prompt).
        let serial_refs: Vec<GenerationResult> = prompts
            .iter()
            .map(|p| gemma4_serial_slot_aware_ref(&load_opts, p, &params))
            .collect();
        let rt = tokio::runtime::Builder::new_multi_thread()
            .worker_threads(8)
            .enable_all()
            .build()
            .expect("rt");

        // Vacuous-test guard: probe both ends of the slot range so neither the
        // low nor the high slots can silently collapse to identical output.
        assert_ne!(
            serial_refs[0].text, serial_refs[1].text,
            "vacuous: prompts 0 and 1 produced identical serial output"
        );
        assert_ne!(
            serial_refs[6].text, serial_refs[7].text,
            "vacuous: prompts 6 and 7 produced identical serial output"
        );

        // Concurrent SlotAware run at the SHIPPED default width (max_slots: 8).
        let loaded_slot = LoadedModel::load(&load_opts).expect("load slot");
        let engine_slot =
            Engine::spawn_with_mode(loaded_slot, 8, None, EngineMode::SlotAware { max_slots: 8 })
                .expect("spawn SlotAware{max_slots:8}");
        let slot_results: Vec<GenerationResult> = rt.block_on(async {
            let mut handles = Vec::new();
            for p in prompts.iter().cloned() {
                let eng = engine_slot.clone();
                let pr = params.clone();
                handles.push(tokio::spawn(async move {
                    eng.generate(p, pr).await.expect("slot generate")
                }));
            }
            let mut out = Vec::new();
            for h in handles {
                out.push(h.await.expect("join"));
            }
            out
        });
        for (i, (slot, serial)) in slot_results.iter().zip(serial_refs.iter()).enumerate() {
            assert_genresult_byte_equal(
                slot,
                serial,
                &format!("ADR-040 iter-F-n8parity — SlotAware N=8 slot {i} vs its serial ref"),
            );
        }
        rt.block_on(engine_slot.shutdown()).expect("shutdown slot");
    }

    /// ADR-040 M-QWEN — qwen35moe serial slot-aware reference (mirror of
    /// [`gemma4_serial_slot_aware_ref_at`] for the Qwen35 architecture): a
    /// single request through `generate_qwen35_once_slot_aware` on a freshly
    /// loaded model with the persistent multi-seq cache provisioned at
    /// `n_seqs`, at a SPECIFIC `slot_id`. The live SlotAware worker decodes
    /// via `Qwen35DecodeState::prefill_seed` + `decode_tick`, which are the
    /// hoisted mirror of this serial fn — byte-parity against these refs is
    /// exactly the F1 mirror-invariant proof (same bar as the gemma4 tests).
    fn qwen35_serial_slot_aware_ref_at(
        load_opts: &LoadOptions,
        prompt: &[u32],
        params: &SamplingParams,
        n_seqs: u32,
        slot_id: SlotId,
    ) -> GenerationResult {
        let mut loaded = LoadedModel::load(load_opts).expect("load qwen35 ref model");
        let LoadedModel::Qwen35(q) = &mut loaded else {
            panic!("qwen35_serial_slot_aware_ref_at: expected a Qwen35 GGUF")
        };
        q.provision_multi_seq_kv_for_slot_aware(n_seqs)
            .expect("provision qwen35 persistent multi-seq KV");
        let mut kv = q
            .persistent_kv_cache
            .take()
            .expect("persistent_kv_cache provisioned");
        let r = crate::serve::api::engine_qwen35::generate_qwen35_once_slot_aware(
            q, prompt, params, None, &mut kv, slot_id,
        )
        .expect("qwen35 serial slot-aware ref");
        q.persistent_kv_cache = Some(kv);
        r
    }

    /// ADR-040 M-QWEN discriminator — serial capacity-invariance pin: the
    /// same prompt, serial, SlotId(0), at n_seqs=1 (per-slot cap = full
    /// 262144) vs n_seqs=8 (per-slot cap = 32768 post-kvcap-split) must be
    /// byte-equal. gemma4 holds this property (its N=8 gate compares
    /// n_seqs=1 serial refs against the max_slots=8 engine and passes).
    /// If THIS pin fails, qwen35's forward output depends on the KV
    /// allocation capacity — a §0.17-class capacity-sensitivity — and the
    /// N=8 parity failure is NOT (necessarily) a concurrency bug.
    #[test]
    fn qwen35_serial_capacity_invariance_pin() {
        if byte_equiv_skip_unless_gated("qwen35_serial_capacity_invariance_pin") {
            return;
        }
        let Ok(gguf) = std::env::var("HF2Q_QWEN35_E2E_GGUF") else {
            eprintln!("[skip] qwen35_serial_capacity_invariance_pin — set HF2Q_QWEN35_E2E_GGUF");
            return;
        };
        std::env::set_var("HF2Q_TQ_KV", "0");
        let load_opts = LoadOptions {
            model_path: PathBuf::from(gguf),
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };
        let prompt = vec![1u32, 2, 3];
        let params = SamplingParams {
            temperature: 0.0,
            max_tokens: 24,
            ..Default::default()
        };
        let r_full = qwen35_serial_slot_aware_ref_at(&load_opts, &prompt, &params, 1, SlotId(0));
        let r_split = qwen35_serial_slot_aware_ref_at(&load_opts, &prompt, &params, 8, SlotId(0));
        eprintln!("[CAPPIN] n_seqs=1 text={:?}", r_full.text);
        eprintln!("[CAPPIN] n_seqs=8 text={:?}", r_split.text);
        assert_genresult_byte_equal(
            &r_split,
            &r_full,
            "ADR-040 M-QWEN — qwen35 serial SlotId(0): n_seqs=8 (cap 32768) vs n_seqs=1 (cap 262144)",
        );
    }

    /// ADR-040 M-QWEN discriminator — SlotAware engine with a SINGLE request
    /// must match the serial slot-aware ref byte-for-byte (loop-faithfulness
    /// of the hoisted `prefill_seed`+`decode_tick` mirror, no concurrency).
    #[test]
    fn qwen35_slot_aware_engine_n1_parity() {
        if byte_equiv_skip_unless_gated("qwen35_slot_aware_engine_n1_parity") {
            return;
        }
        let Ok(gguf) = std::env::var("HF2Q_QWEN35_E2E_GGUF") else {
            eprintln!("[skip] qwen35_slot_aware_engine_n1_parity — set HF2Q_QWEN35_E2E_GGUF");
            return;
        };
        std::env::set_var("HF2Q_TQ_KV", "0");
        let load_opts = LoadOptions {
            model_path: PathBuf::from(gguf),
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };
        let prompt = vec![1u32, 2, 3];
        let params = SamplingParams {
            temperature: 0.0,
            max_tokens: 24,
            ..Default::default()
        };
        let serial = qwen35_serial_slot_aware_ref_at(&load_opts, &prompt, &params, 1, SlotId(0));
        let loaded = LoadedModel::load(&load_opts).expect("load qwen35 n1");
        let engine =
            Engine::spawn_with_mode(loaded, 8, None, EngineMode::SlotAware { max_slots: 8 })
                .expect("spawn qwen35 SlotAware n1");
        let rt = tokio::runtime::Builder::new_multi_thread()
            .worker_threads(2)
            .enable_all()
            .build()
            .expect("rt");
        let r = rt
            .block_on(engine.clone().generate(prompt, params))
            .expect("qwen35 n1 generate");
        eprintln!("[N1PIN] engine text={:?}", r.text);
        eprintln!("[N1PIN] serial text={:?}", serial.text);
        assert_genresult_byte_equal(
            &r,
            &serial,
            "ADR-040 M-QWEN — qwen35 SlotAware engine N=1 vs serial slot-aware ref",
        );
        rt.block_on(engine.shutdown()).expect("shutdown");
    }

    /// ADR-040 M-QWEN (closes the §0.12 tracked-open item, 2026-07-01):
    /// qwen35moe cross-slot correctness was "correct-by-construction"
    /// (stateless forward, per-slot cursor in `current_len[slot]`) but
    /// empirically UNPROVEN — no qwen35 GGUF was staged when the gemma4
    /// N=8 parity gates landed. This is the direct mirror of
    /// [`slot_aware_n8_per_slot_parity_vs_serial`] for qwen35moe: 8 distinct
    /// greedy prompts concurrently through one `SlotAware { max_slots: 8 }`
    /// engine, each asserted byte-identical to its independent serial
    /// slot-aware reference; plus a slot-equivalence pin (same prompt serial
    /// at SlotId(0) vs SlotId(7) byte-equal — pins per-slot KV region
    /// indexing in isolation). Gated separately from the gemma4 tests:
    /// `HF2Q_BYTE_EQUIV_E2E=1` + `HF2Q_QWEN35_E2E_GGUF=<path>` (codex
    /// review 2026-07-01: do NOT overload the gemma-oriented
    /// `HF2Q_BYTE_EQUIV_E2E_GGUF`).
    #[test]
    fn slot_aware_qwen35_n8_per_slot_parity_vs_serial() {
        if byte_equiv_skip_unless_gated("slot_aware_qwen35_n8_per_slot_parity_vs_serial") {
            return;
        }
        let Ok(gguf) = std::env::var("HF2Q_QWEN35_E2E_GGUF") else {
            eprintln!(
                "[skip] slot_aware_qwen35_n8_per_slot_parity_vs_serial — set \
                 HF2Q_QWEN35_E2E_GGUF=<qwen35moe gguf> to run (ADR-040 M-QWEN)"
            );
            return;
        };
        // Multi-slot qwen35 REQUIRES the F32 full-attn KV path: slot_id>0
        // with TQ-active KV is the typed Phase B4a-cont deferral
        // ("TQ encode + TQ SDPA kernels are not yet slot-aware", ADR-040
        // §6.1.5/§6.1.6, dossier R5) and fails CLOSED — empirically
        // confirmed by this test's first run 2026-07-01 (build_gated_attn_layer
        // slot_id=7 typed error). HF2Q_TQ_KV defaults ON, so pin it OFF for
        // the supported multi-slot configuration this gate proves.
        std::env::set_var("HF2Q_TQ_KV", "0");
        let gguf_path = PathBuf::from(gguf);
        assert!(
            gguf_path.exists(),
            "qwen35 GGUF missing: {}",
            gguf_path.display()
        );
        let load_opts = LoadOptions {
            model_path: gguf_path,
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };

        // Eight distinct greedy prompts (same shape as the gemma4 N=8 gate).
        let prompts: Vec<Vec<u32>> = vec![
            vec![1, 2, 3],
            vec![4, 5, 6, 7],
            vec![8, 9],
            vec![10, 11, 12, 13, 14],
            vec![15, 16],
            vec![17, 18, 19, 20],
            vec![21, 22, 23],
            vec![24, 25, 26, 27, 28],
        ];
        let params = SamplingParams {
            temperature: 0.0,
            max_tokens: 24,
            ..Default::default()
        };

        // Slot-equivalence pin: same prompt, serial, SlotId(0) vs SlotId(7)
        // on an n_seqs=8 provisioning — pins per-slot KV region indexing.
        let pin0 = qwen35_serial_slot_aware_ref_at(&load_opts, &prompts[0], &params, 8, SlotId(0));
        let pin7 = qwen35_serial_slot_aware_ref_at(&load_opts, &prompts[0], &params, 8, SlotId(7));
        assert_genresult_byte_equal(
            &pin7,
            &pin0,
            "ADR-040 M-QWEN — same prompt serial SlotId(7) vs SlotId(0)",
        );

        // Serial slot-aware references (one fresh model per prompt).
        let serial_refs: Vec<GenerationResult> = prompts
            .iter()
            .map(|p| qwen35_serial_slot_aware_ref_at(&load_opts, p, &params, 1, SlotId(0)))
            .collect();

        // Vacuous-test guard at both ends of the slot range.
        assert_ne!(
            serial_refs[0].text, serial_refs[1].text,
            "vacuous: qwen35 prompts 0 and 1 produced identical serial output"
        );
        assert_ne!(
            serial_refs[6].text, serial_refs[7].text,
            "vacuous: qwen35 prompts 6 and 7 produced identical serial output"
        );

        let rt = tokio::runtime::Builder::new_multi_thread()
            .worker_threads(8)
            .enable_all()
            .build()
            .expect("rt");

        // Concurrent SlotAware run at max_slots=8.
        let loaded_slot = LoadedModel::load(&load_opts).expect("load qwen35 slot");
        let engine_slot =
            Engine::spawn_with_mode(loaded_slot, 8, None, EngineMode::SlotAware { max_slots: 8 })
                .expect("spawn qwen35 SlotAware{max_slots:8}");
        let slot_results: Vec<GenerationResult> = rt.block_on(async {
            let mut handles = Vec::new();
            for p in prompts.iter().cloned() {
                let eng = engine_slot.clone();
                let pr = params.clone();
                handles.push(tokio::spawn(async move {
                    eng.generate(p, pr).await.expect("qwen35 slot generate")
                }));
            }
            let mut out = Vec::new();
            for h in handles {
                out.push(h.await.expect("join"));
            }
            out
        });
        for (i, (slot, serial)) in slot_results.iter().zip(serial_refs.iter()).enumerate() {
            assert_genresult_byte_equal(
                slot,
                serial,
                &format!("ADR-040 M-QWEN — qwen35 SlotAware N=8 slot {i} vs its serial ref"),
            );
        }
        rt.block_on(engine_slot.shutdown())
            .expect("shutdown qwen35 slot");
    }

    /// ADR-040 §0.19 — HIGH-RATE long-prompt determinism REPRO. Runs the
    /// concurrent N=8 SlotAware batch `repeats` times over LONG prompts (each
    /// > 512 KV ⇒ split-K `nwg=32`, max tmp+reduce contention — vs the
    /// short-prompt nwg=16 parity test) and asserts every repeat is byte-equal
    /// to repeat 0, per slot. Self-consistency (no serial ref): directly
    /// measures the §0.19 batched-FA non-determinism and AMPLIFIES it via
    /// nwg=32 to beat the batch-to-batch variance that made the short-prompt
    /// ×30 diagnostic inconclusive. Simultaneous admission ⇒ all slots share
    /// the `same_bucket` PATH A batched split-K flash (the suspect path).
    /// Env: HF2Q_S019_PROMPT_LEN (600), HF2Q_S019_REPEATS (8),
    /// HF2Q_S019_MAXTOK (32). Set HF2Q_HYBRID_NWG=1 to force split-K OFF —
    /// THE DISCRIMINATOR: nwg=1 clean + default flakes ⇒ split-K tmp+reduce is
    /// §0.19's root; both flake ⇒ split-K exonerated, root is upstream
    /// (prefill FA / KV cache). Prints `§0.19 REPRO: F/T mismatches`.
    #[test]
    fn slot_aware_n8_long_prompt_s019_determinism_repro() {
        if byte_equiv_skip_unless_gated("slot_aware_n8_long_prompt_s019_determinism_repro") {
            return;
        }
        let gguf_path: PathBuf = std::env::var(BYTE_EQUIV_E2E_GGUF_ENV)
            .map(PathBuf::from)
            .expect("HF2Q_BYTE_EQUIV_E2E_GGUF set");
        assert!(gguf_path.exists(), "GGUF missing: {}", gguf_path.display());
        let load_opts = LoadOptions {
            model_path: gguf_path.clone(),
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };

        let prompt_len: usize = std::env::var("HF2Q_S019_PROMPT_LEN")
            .ok()
            .and_then(|s| s.parse().ok())
            .unwrap_or(600);
        let repeats: usize = std::env::var("HF2Q_S019_REPEATS")
            .ok()
            .and_then(|s| s.parse().ok())
            .unwrap_or(8);
        let max_tokens: usize = std::env::var("HF2Q_S019_MAXTOK")
            .ok()
            .and_then(|s| s.parse().ok())
            .unwrap_or(32);
        // Number of concurrent sequences (default 8 = the shipped batch width).
        // Set HF2Q_S019_NSEQ=1 to isolate single-stream decode (apples-to-apples
        // with the llama-completion single-seq determinism control).
        let nseq: u32 = std::env::var("HF2Q_S019_NSEQ")
            .ok()
            .and_then(|s| s.parse().ok())
            .filter(|&n| (1..=8).contains(&n))
            .unwrap_or(8);

        // `nseq` DISTINCT long prompts, each `prompt_len` tokens from a disjoint
        // token-id band (well within gemma's ~256k vocab; distinct ⇒ distinct
        // output for the vacuous guard).
        let prompts: Vec<Vec<u32>> = (0..nseq)
            .map(|s| {
                let base = 100u32 + s * (prompt_len as u32 + 8);
                (0..prompt_len as u32).map(|t| base + t).collect()
            })
            .collect();
        let params = SamplingParams {
            temperature: 0.0,
            max_tokens,
            ..Default::default()
        };

        let rt = tokio::runtime::Builder::new_multi_thread()
            .worker_threads(8)
            .enable_all()
            .build()
            .expect("rt");

        let loaded_slot = LoadedModel::load(&load_opts).expect("load slot");
        let engine_slot =
            Engine::spawn_with_mode(loaded_slot, 8, None, EngineMode::SlotAware { max_slots: 8 })
                .expect("spawn SlotAware{max_slots:8}");

        let run_batch = |eng: &Engine| -> Vec<String> {
            rt.block_on(async {
                let mut handles = Vec::new();
                for p in prompts.iter().cloned() {
                    let e = eng.clone();
                    let pr = params.clone();
                    handles.push(tokio::spawn(async move {
                        e.generate(p, pr).await.expect("slot generate")
                    }));
                }
                let mut out = Vec::new();
                for h in handles {
                    out.push(h.await.expect("join").text);
                }
                out
            })
        };

        let ref_run = run_batch(&engine_slot);
        // Vacuous guard: distinct prompts must give distinct output (≥2 seqs).
        if ref_run.len() >= 2 {
            assert_ne!(
                ref_run[0], ref_run[1],
                "vacuous: slots 0 and 1 produced identical output"
            );
        }

        let mut flakes = 0usize;
        for r in 1..repeats {
            let run = run_batch(&engine_slot);
            for (i, (cur, base)) in run.iter().zip(ref_run.iter()).enumerate() {
                if cur != base {
                    flakes += 1;
                    let n = cur.len().min(base.len());
                    let div = (0..n).find(|&k| cur.as_bytes()[k] != base.as_bytes()[k]);
                    eprintln!(
                        "§0.19 FLAKE repeat {r} slot {i}: first-divergent byte {:?} (len {} vs ref {})",
                        div,
                        cur.len(),
                        base.len()
                    );
                }
            }
        }
        let total = (repeats - 1) * 8;
        eprintln!(
            "§0.19 REPRO: {flakes}/{total} per-(repeat,slot) mismatches | repeats={repeats} prompt_len={prompt_len} max_tokens={max_tokens} HYBRID_NWG={}",
            std::env::var("HF2Q_HYBRID_NWG").unwrap_or_else(|_| "default".into())
        );
        rt.block_on(engine_slot.shutdown()).expect("shutdown slot");
        assert_eq!(
            flakes, 0,
            "§0.19 non-determinism: {flakes}/{total} mismatches"
        );
    }

    /// ADR-040 iter-G(a) — cross-slot batched-prefill FORWARD isolation gate.
    /// Concatenates N distinct prompts into ONE forward pass via
    /// `forward_prefill_batched_multi_seq` and asserts each seq's FIRST decode
    /// token equals that prompt prefilled ALONE (single-seq
    /// `forward_prefill_batched`). This validates, on a real model, the
    /// block-diagonal mask isolation + per-seq RoPE positions + N-row head
    /// (deltas 1, 2, 4) independently of the admit loop. Delta 3 (KV scatter,
    /// decode-only) is gated by the E2E full-generation parity test. Same
    /// HF2Q_BYTE_EQUIV_E2E gate as the serial/parity harness.
    #[test]
    fn iter_g_a_multi_seq_prefill_first_token_isolation() {
        if byte_equiv_skip_unless_gated("iter_g_a_multi_seq_prefill_first_token_isolation") {
            return;
        }
        let gguf_path: PathBuf = std::env::var(BYTE_EQUIV_E2E_GGUF_ENV)
            .map(PathBuf::from)
            .expect("HF2Q_BYTE_EQUIV_E2E_GGUF set");
        assert!(gguf_path.exists(), "GGUF missing: {}", gguf_path.display());
        let load_opts = LoadOptions {
            model_path: gguf_path,
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };

        // N=1 mechanism gate: a single 70-token prompt exercising the full
        // multi-seq machinery (GPU block-diagonal mask builder, per-seq RoPE
        // positions, N-row head, per-slot KV scatter) at offset 0 — which is
        // offset-invariance-aligned, so it matches the single-seq reference.
        //
        // NOTE: N>1 isolation is NOT yet byte-identical — the orchestration
        // (GPU mask, positions, KV, head) is CORRECT (N=1 exact; N=8 BF16
        // tensor-mm = 7/8 byte-exact), but the ATTENTION KERNELS' handling of
        // block-diagonal masks blocks full byte-identity:
        //   - F16 FA (D256+D512): sequence-offset-NON-invariant with block-diag
        //     masks (fork-bisected; same family as task #19);
        //   - BF16 D512 FA: §0.19 enumeration-coherence bug (worse isolation);
        //   - tensor-mm globals: closest (7/8), residual = FP-accumulation over
        //     MASKED columns (matmul sums all T cols → near-tie argmax flips vs
        //     the per-seq single-seq sum — fundamental to matmul attention).
        // True byte-identity needs a D512 FA kernel that SKIPS masked tiles AND
        // is offset-invariant (only FA skips masked tiles; matmul can't) — i.e.
        // fixing the D512 FA kernel (converges task #19). Set HF2Q_ITERGA_N8=1
        // for the 8-prompt isolation matrix. See ADR-040 §0.20.
        let prompts: Vec<Vec<u32>> = if std::env::var("HF2Q_ITERGA_N8").as_deref() == Ok("1") {
            // 64-token prompts → each seq starts at a 64-multiple offset =
            // D512 chunk(C=64)-aligned, so post-blk-fix F16 FA isolates byte-exact.
            (0..8u32)
                .map(|i| {
                    (0..64u32)
                        .map(|j| 1 + (i.wrapping_mul(131).wrapping_add(j.wrapping_mul(7)) % 4000))
                        .collect()
                })
                .collect()
        } else {
            vec![(0..70u32).map(|j| 1 + (j.wrapping_mul(7) % 4000)).collect()]
        };
        let n = prompts.len();
        let max_decode = 24usize;

        // Single-seq references: each prompt's FIRST decode token, prefilled
        // ALONE through the production-default batched prefill.
        let mut ref_tokens: Vec<u32> = Vec::with_capacity(n);
        {
            let mut loaded = LoadedModel::load(&load_opts).expect("load ref model");
            let LoadedModel::Gemma(g) = &mut loaded else {
                panic!("iter-G(a) test: expected a Gemma GGUF")
            };
            for p in &prompts {
                let t = g
                    .weights
                    .forward_prefill_batched(p, max_decode, 0, &mut g.ctx)
                    .expect("single-seq ref prefill");
                ref_tokens.push(t);
            }
        }
        // Vacuous-test guard: the refs must not all collapse to one token.
        if n > 1 {
            assert!(
                ref_tokens.iter().any(|&t| t != ref_tokens[0]),
                "vacuous: all single-seq first tokens identical ({:?})",
                ref_tokens
            );
        }

        // Multi-seq cross-slot prefill — all N prompts in ONE forward.
        let mut loaded = LoadedModel::load(&load_opts).expect("load multi-seq model");
        let LoadedModel::Gemma(g) = &mut loaded else {
            panic!("iter-G(a) test: expected a Gemma GGUF")
        };
        g.provision_multi_seq_kv_for_slot_aware(n as u32)
            .expect("provision multi-seq KV");
        let scaffold = g
            .multi_seq_kv_hybrid
            .take()
            .expect("hybrid scaffold present (HF2Q_HYBRID_KV default-on)");
        let seqs: Vec<(Vec<u32>, SlotId)> = prompts
            .iter()
            .enumerate()
            .map(|(i, p)| (p.clone(), SlotId(i as u32)))
            .collect();
        let ms_tokens = g
            .weights
            .forward_prefill_batched_multi_seq(&seqs, &scaffold, max_decode, &mut g.ctx)
            .expect("multi-seq cross-slot prefill");
        g.multi_seq_kv_hybrid = Some(scaffold);

        assert_eq!(ms_tokens.len(), n, "multi-seq returned wrong token count");
        let mut mismatches = 0usize;
        for i in 0..n {
            let ok = ms_tokens[i] == ref_tokens[i];
            if !ok {
                mismatches += 1;
            }
            eprintln!(
                "[iter-G(a) seq {i}] cross-slot={} single-seq={} {}",
                ms_tokens[i],
                ref_tokens[i],
                if ok { "OK" } else { "MISMATCH" },
            );
        }
        assert_eq!(
            mismatches, 0,
            "iter-G(a) {mismatches}/{n} seqs diverged (block-diagonal isolation broken)"
        );
        eprintln!("[iter-G(a)] first-token isolation PASS: {n} seqs, tokens={ms_tokens:?}");
    }

    /// ADR-040 iter-G(a) — BF16 cross-slot prefill DETERMINISM + ISOLATION gate
    /// (codex's hard gate before committing to the BF16 batched path). Multi-seq
    /// prefill runs on BF16 FA (the F16 FA prefill kernels are the uncracked
    /// §0.19 heisenbug on block-diagonal masks). Demands, under the real
    /// allocator/buffer-reuse (K repeated in-process runs):
    ///   1. DETERMINISM — K runs of the identical N=8 varied-length batch produce
    ///      BYTE-IDENTICAL first tokens (run-to-run; the user-facing requirement).
    ///   2. ISOLATION — each seq matches its single-seq BF16 reference.
    ///   3. NO CONTAMINATION — a sequence held constant while its batch-mates are
    ///      replaced produces the same token (cross-seq independence).
    /// Run with HF2Q_FA_F16=0 so the single-seq refs are BF16 too (multi-seq
    /// forces BF16 regardless). FAIL ON ANY FLAKE — one divergence means BF16 is
    /// only a lower-probability manifestation, not a fix. HF2Q_ITERGA_KRUNS sets K.
    #[test]
    fn iter_g_a_bf16_determinism_isolation_gate() {
        if byte_equiv_skip_unless_gated("iter_g_a_bf16_determinism_isolation_gate") {
            return;
        }
        let gguf_path: PathBuf = std::env::var(BYTE_EQUIV_E2E_GGUF_ENV)
            .map(PathBuf::from)
            .expect("HF2Q_BYTE_EQUIV_E2E_GGUF set");
        assert!(gguf_path.exists(), "GGUF missing: {}", gguf_path.display());
        let load_opts = LoadOptions {
            model_path: gguf_path,
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };
        let lens = [26u32, 40, 13, 55, 70, 19, 33, 48];
        let mk = |i: u32, l: u32| -> Vec<u32> {
            (0..l)
                .map(|j| 1 + (i.wrapping_mul(131).wrapping_add(j.wrapping_mul(7)) % 4000))
                .collect()
        };
        let prompts: Vec<Vec<u32>> = (0..8u32).map(|i| mk(i, lens[i as usize])).collect();
        let n = prompts.len();
        let max_decode = 24usize;
        let k_runs: usize = std::env::var("HF2Q_ITERGA_KRUNS")
            .ok()
            .and_then(|v| v.parse().ok())
            .unwrap_or(20);

        let mut loaded = LoadedModel::load(&load_opts).expect("load");
        let LoadedModel::Gemma(g) = &mut loaded else {
            panic!("expected Gemma GGUF")
        };
        g.provision_multi_seq_kv_for_slot_aware(n as u32)
            .expect("provision multi-seq KV");

        let run_batch = |g: &mut GemmaLoadedModel, batch: &[Vec<u32>]| -> Vec<u32> {
            let seqs: Vec<(Vec<u32>, SlotId)> = batch
                .iter()
                .enumerate()
                .map(|(i, p)| (p.clone(), SlotId(i as u32)))
                .collect();
            let scaffold = g.multi_seq_kv_hybrid.take().expect("hybrid scaffold");
            let toks = g
                .weights
                .forward_prefill_batched_multi_seq(&seqs, &scaffold, max_decode, &mut g.ctx)
                .expect("multi-seq prefill");
            g.multi_seq_kv_hybrid = Some(scaffold);
            toks
        };

        // 1. DETERMINISM — k_runs repeated identical batches.
        let mut runs: Vec<Vec<u32>> = Vec::with_capacity(k_runs);
        for _ in 0..k_runs {
            runs.push(run_batch(g, &prompts));
        }
        for k in 1..k_runs {
            assert_eq!(
                runs[k], runs[0],
                "DETERMINISM FAIL @run {k}: BF16 multi-seq is NOT a fix (heisenbug survives)\n run0={:?}\n run{k}={:?}",
                runs[0], runs[k]
            );
        }
        eprintln!(
            "[iter-G(a) BF16] DETERMINISM {k_runs}/{k_runs} byte-identical: {:?}",
            runs[0]
        );

        // 2. NO CONTENT LEAKAGE (the correct isolation bar — NOT byte-identity
        //    to the single-seq path, which is the accepted-benign batched-vs-
        //    serial FP gap per ADR §B1/AC4=(b)). For each seq i: rebuild the
        //    batch with seq i UNCHANGED but every OTHER seq's CONTENT reseeded
        //    (same lengths, same positions). seq i's token MUST be unchanged —
        //    if it depended on a neighbor's CONTENT, that is real leakage.
        //    (Measured separately: B's token is byte-invariant to A's content.)
        for hold in 0..n {
            let mut batch: Vec<Vec<u32>> = Vec::with_capacity(n);
            for i in 0..n {
                if i == hold {
                    batch.push(prompts[i].clone());
                } else {
                    // reseed content, SAME length + position as prompts[i].
                    batch.push(mk(i as u32 + 7000 + hold as u32, lens[i]));
                }
            }
            let r = run_batch(g, &batch);
            assert_eq!(
                r[hold], runs[0][hold],
                "LEAKAGE: seq {hold} token changed ({} -> {}) when OTHER seqs' content changed \
                 (lengths/positions held) — real cross-sequence contamination",
                runs[0][hold], r[hold]
            );
        }
        eprintln!("[iter-G(a) BF16] NO-LEAKAGE: all {n} seqs invariant to batch-mates' content");

        // 3. INFORMATIONAL — how far the benign FP gap moves vs single-seq (B1).
        //    NOT an assertion: batched != serial by a benign near-tie margin.
        let mut refs: Vec<u32> = Vec::with_capacity(n);
        for p in &prompts {
            refs.push(
                g.weights
                    .forward_prefill_batched(p, max_decode, 0, &mut g.ctx)
                    .expect("ref"),
            );
        }
        let benign = (0..n).filter(|&i| runs[0][i] != refs[i]).count();
        eprintln!(
            "[iter-G(a) BF16] benign batched-vs-serial FP gap: {benign}/{n} near-tie flips \
             (accepted per ADR §B1/AC4=(b); determinism + no-leakage are the bar)"
        );
    }

    /// ADR-040 iter-G(a) — E2E batched-admit + N=8 TTFT. Drives the FULL
    /// production path: 8 concurrent greedy Generate requests through the
    /// SlotAware engine with HF2Q_CROSS_SLOT_ADMIT=1 → the admit loop batches
    /// them into ONE multi-seq prefill. Asserts (1) all 8 complete with output,
    /// (2) the batched path actually fired (ITER_GA_BATCHED_ADMIT_COUNT>0). Then
    /// measures wall-time for 8 concurrent max_tokens=1 requests with batching
    /// ON vs OFF — the prefill TTFT lever (§0.17). Greedy + BF16 multi-seq.
    #[test]
    fn iter_g_a_batched_admit_e2e_and_ttft() {
        if byte_equiv_skip_unless_gated("iter_g_a_batched_admit_e2e_and_ttft") {
            return;
        }
        let gguf_path: PathBuf = std::env::var(BYTE_EQUIV_E2E_GGUF_ENV)
            .map(PathBuf::from)
            .expect("HF2Q_BYTE_EQUIV_E2E_GGUF set");
        assert!(gguf_path.exists(), "GGUF missing: {}", gguf_path.display());
        let load_opts = LoadOptions {
            model_path: gguf_path,
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };
        let prompts: Vec<Vec<u32>> = (0..8u32)
            .map(|i| {
                (0..(20 + i * 3))
                    .map(|j| 1 + (i.wrapping_mul(131).wrapping_add(j.wrapping_mul(7)) % 4000))
                    .collect()
            })
            .collect();
        let rt = tokio::runtime::Builder::new_multi_thread()
            .worker_threads(8)
            .enable_all()
            .build()
            .expect("rt");

        // multi-seq forces BF16 regardless; set it so single-seq peers match.
        std::env::set_var("HF2Q_FA_F16", "0");

        // ── E2E: batched admit ON, 8 concurrent greedy generates ──────────
        std::env::set_var("HF2Q_CROSS_SLOT_ADMIT", "1");
        ITER_GA_BATCHED_ADMIT_COUNT.store(0, std::sync::atomic::Ordering::Relaxed);
        let params = SamplingParams {
            temperature: 0.0,
            max_tokens: 24,
            ..Default::default()
        };
        let loaded = LoadedModel::load(&load_opts).expect("load e2e");
        let engine =
            Engine::spawn_with_mode(loaded, 16, None, EngineMode::SlotAware { max_slots: 8 })
                .expect("spawn SlotAware");
        let results: Vec<GenerationResult> = rt.block_on(async {
            let mut handles = Vec::new();
            for p in prompts.iter().cloned() {
                let eng = engine.clone();
                let pr = params.clone();
                handles.push(tokio::spawn(async move {
                    eng.generate(p, pr).await.expect("generate")
                }));
            }
            let mut out = Vec::new();
            for h in handles {
                out.push(h.await.expect("join"));
            }
            out
        });
        for (i, r) in results.iter().enumerate() {
            assert!(
                !r.text.is_empty(),
                "iter-G(a) E2E: seq {i} produced empty output"
            );
            assert!(r.completion_tokens > 0, "iter-G(a) E2E: seq {i} no tokens");
        }
        let batched = ITER_GA_BATCHED_ADMIT_COUNT.load(std::sync::atomic::Ordering::Relaxed);
        assert!(batched > 0, "iter-G(a) E2E: batched admit NEVER fired (count=0) — the multi-seq prefill path was not exercised");
        eprintln!("[iter-G(a) E2E] 8/8 complete; batched-admit forwards fired = {batched}");
        rt.block_on(engine.shutdown()).expect("shutdown e2e");

        // ── TTFT A/B: 8 concurrent max_tokens=1, batched ON vs OFF ────────
        let ttft_params = SamplingParams {
            temperature: 0.0,
            max_tokens: 1,
            ..Default::default()
        };
        let measure = |on: bool| -> f64 {
            if on {
                std::env::set_var("HF2Q_CROSS_SLOT_ADMIT", "1");
            } else {
                std::env::remove_var("HF2Q_CROSS_SLOT_ADMIT");
            }
            let loaded = LoadedModel::load(&load_opts).expect("load ttft");
            let engine =
                Engine::spawn_with_mode(loaded, 16, None, EngineMode::SlotAware { max_slots: 8 })
                    .expect("spawn ttft");
            // warm-up (pipeline bake) — not timed.
            let _ = rt.block_on(
                engine
                    .clone()
                    .generate(prompts[0].clone(), ttft_params.clone()),
            );
            let t0 = std::time::Instant::now();
            let _: Vec<GenerationResult> = rt.block_on(async {
                let mut handles = Vec::new();
                for p in prompts.iter().cloned() {
                    let eng = engine.clone();
                    let pr = ttft_params.clone();
                    handles.push(tokio::spawn(async move {
                        eng.generate(p, pr).await.expect("gen")
                    }));
                }
                let mut out = Vec::new();
                for h in handles {
                    out.push(h.await.expect("join"));
                }
                out
            });
            let dt = t0.elapsed().as_secs_f64() * 1000.0;
            rt.block_on(engine.shutdown()).expect("shutdown ttft");
            dt
        };
        let t_on = measure(true);
        let t_off = measure(false);
        eprintln!(
            "[iter-G(a) TTFT] 8 concurrent (max_tokens=1): batched-ON {t_on:.0} ms vs sequential-OFF {t_off:.0} ms — speedup {:.2}x",
            t_off / t_on.max(0.001),
        );
        std::env::remove_var("HF2Q_CROSS_SLOT_ADMIT");
    }

    /// ADR-040 iter-G(a) DIAGNOSTIC — bisect the offset-mod-4 isolation bug.
    /// Runs a single-seq forward of prompt B (offset 0), then a multi-seq
    /// forward of [A, B] where A's length puts B at offset ≡2 mod 4. With
    /// HF2Q_CKSUM_PERSEQ=1 every layer prints a per-seq pf_hidden FNV — compare
    /// `[ROWCK single L..]` against `[ROWCK multi.s1 L..]` to find the FIRST
    /// divergent layer (sliding=D256 vs global=D512). Diagnostic only.
    #[test]
    fn iter_g_a_bisect_offset() {
        if byte_equiv_skip_unless_gated("iter_g_a_bisect_offset") {
            return;
        }
        let gguf_path: PathBuf = std::env::var(BYTE_EQUIV_E2E_GGUF_ENV)
            .map(PathBuf::from)
            .expect("HF2Q_BYTE_EQUIV_E2E_GGUF set");
        let load_opts = LoadOptions {
            model_path: gguf_path,
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };
        // A len configurable via HF2Q_BISECT_ALEN (default 2 → B offset 2 ≡2 mod4).
        // B len via HF2Q_BISECT_BLEN (default 10). Use larger to hit tensor-mm (>64).
        let alen: usize = std::env::var("HF2Q_BISECT_ALEN")
            .ok()
            .and_then(|v| v.parse().ok())
            .unwrap_or(2);
        let blen: usize = std::env::var("HF2Q_BISECT_BLEN")
            .ok()
            .and_then(|v| v.parse().ok())
            .unwrap_or(10);
        // HF2Q_BISECT_ASEED varies A's CONTENT at fixed length — the decisive
        // leakage-vs-benign-FP discriminator: if B's token is invariant to A's
        // content (B depends only on A's presence/length), there is NO leakage.
        let aseed: u32 = std::env::var("HF2Q_BISECT_ASEED")
            .ok()
            .and_then(|v| v.parse().ok())
            .unwrap_or(0);
        let prompt_a: Vec<u32> = (0..alen as u32)
            .map(|j| 100 + j + aseed.wrapping_mul(311) % 30000)
            .collect();
        let prompt_b: Vec<u32> = (0..blen as u32)
            .map(|j| 1 + j.wrapping_mul(7) % 4000)
            .collect();
        let max_decode = 24usize;

        eprintln!("[BISECT] ===SINGLEA=== single-seq forward of prompt A (offset 0, len {alen})");
        let single_a_tok = {
            let mut loaded = LoadedModel::load(&load_opts).expect("load singleA");
            let LoadedModel::Gemma(g) = &mut loaded else {
                panic!("expected Gemma")
            };
            g.weights
                .forward_prefill_batched(&prompt_a, max_decode, 0, &mut g.ctx)
                .expect("singleA forward")
        };
        eprintln!("[BISECT] singleA first_token={single_a_tok}");

        eprintln!("[BISECT] ===SINGLEB=== single-seq forward of prompt B (offset 0, len {blen})");
        let single_tok = {
            let mut loaded = LoadedModel::load(&load_opts).expect("load single");
            let LoadedModel::Gemma(g) = &mut loaded else {
                panic!("expected Gemma")
            };
            g.weights
                .forward_prefill_batched(&prompt_b, max_decode, 0, &mut g.ctx)
                .expect("single forward")
        };
        eprintln!("[BISECT] single B first_token={single_tok}");

        eprintln!("[BISECT] === multi-seq forward of [A(len {alen}), B(len {blen})] — B at offset {alen} ===");
        let multi_toks = {
            let mut loaded = LoadedModel::load(&load_opts).expect("load multi");
            let LoadedModel::Gemma(g) = &mut loaded else {
                panic!("expected Gemma")
            };
            g.provision_multi_seq_kv_for_slot_aware(2)
                .expect("provision");
            let scaffold = g.multi_seq_kv_hybrid.take().expect("scaffold");
            let seqs: Vec<(Vec<u32>, SlotId)> =
                vec![(prompt_a.clone(), SlotId(0)), (prompt_b.clone(), SlotId(1))];
            let toks = g
                .weights
                .forward_prefill_batched_multi_seq(&seqs, &scaffold, max_decode, &mut g.ctx)
                .expect("multi forward");
            g.multi_seq_kv_hybrid = Some(scaffold);
            toks
        };
        eprintln!(
            "[BISECT] multi tokens={multi_toks:?}; singleA={single_a_tok} seq0(A) {}; \
             singleB={single_tok} seq1(B) {}",
            if multi_toks.first() == Some(&single_a_tok) {
                "=="
            } else {
                "!="
            },
            if multi_toks.get(1) == Some(&single_tok) {
                "=="
            } else {
                "!="
            },
        );
    }

    /// ADR-040 iter-I — the vectorizable `argmax_f32_first_max` must be
    /// BYTE-IDENTICAL to the original scalar first-max `v > bv` loop on every
    /// input shape (random, ties, -inf, NaN, empty, single). This is the
    /// byte-identity gate for replacing the decode-critical-path argmax.
    #[test]
    fn argmax_f32_first_max_matches_scalar_ref() {
        // The exact original scalar reference (pre-iter-I).
        fn scalar_ref(xs: &[f32]) -> (u32, f32) {
            let mut bi = 0usize;
            let mut bv = f32::NEG_INFINITY;
            for (i, &v) in xs.iter().enumerate() {
                if v > bv {
                    bv = v;
                    bi = i;
                }
            }
            (bi as u32, bv)
        }
        let mut cases: Vec<Vec<f32>> = Vec::new();
        // Deterministic pseudo-random rows of vocab-like length, plus edges.
        let mut s: u64 = 0x1234_5678_9abc_def0;
        for len in [0usize, 1, 2, 7, 256, 1024, 262144] {
            let mut row = Vec::with_capacity(len);
            for _ in 0..len {
                s = s
                    .wrapping_mul(6364136223846793005)
                    .wrapping_add(1442695040888963407);
                row.push(((s >> 33) as f32) / (u32::MAX as f32) - 0.5);
            }
            cases.push(row);
        }
        // Tie cases: first-max must win.
        cases.push(vec![1.0, 2.0, 2.0, 2.0, 1.0]); // max 2.0 first at idx 1
        cases.push(vec![5.0, 5.0, 5.0]); // all equal -> idx 0
        cases.push(vec![f32::NEG_INFINITY; 4]); // all -inf -> (0, -inf)
        cases.push(vec![-1.0, f32::NAN, -2.0, -0.5]); // NaN skipped -> idx 3 (-0.5)
        cases.push(vec![f32::NAN, f32::NAN]); // all NaN -> (0, -inf)
        cases.push(vec![0.0, -0.0, 0.0]); // +0/-0: 0.0 not > 0.0 -> idx 0
        for (ci, c) in cases.iter().enumerate() {
            let r = scalar_ref(c);
            let g = argmax_f32_first_max(c);
            assert_eq!(
                r,
                g,
                "argmax mismatch case {ci} (len {}): scalar {r:?} vs fast {g:?}",
                c.len()
            );
        }
    }

    /// ADR-040 M2.2 — INFORMATIONAL throughput probe for the `[N,hidden]`
    /// batched decode body (S2/S3). Drives 4 concurrent SlotAware generates of
    /// `BENCH_TOKENS` tokens and prints aggregate decode tok/s. Read the env
    /// `HF2Q_BATCHED_BODY` the harness was launched with and run it BOTH ways to
    /// compare. Gated by HF2Q_BATCHED_BENCH=1 (+ the shared E2E GGUF gate); not
    /// an assertion — the parity bar is owned by `slot_aware_n4`.
    #[test]
    fn slot_aware_n4_batched_body_throughput_probe() {
        if std::env::var("HF2Q_BATCHED_BENCH").as_deref() != Ok("1") {
            eprintln!("[skip] slot_aware_n4_batched_body_throughput_probe — set HF2Q_BATCHED_BENCH=1 + HF2Q_BYTE_EQUIV_E2E_GGUF to run");
            return;
        }
        let gguf_path: PathBuf = std::env::var(BYTE_EQUIV_E2E_GGUF_ENV)
            .map(PathBuf::from)
            .expect("HF2Q_BYTE_EQUIV_E2E_GGUF set");
        assert!(gguf_path.exists(), "GGUF missing: {}", gguf_path.display());
        let load_opts = LoadOptions {
            model_path: gguf_path,
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };
        // HF2Q_BENCH_TOKENS = decode length per stream (default 128).
        let bench_tokens: usize = std::env::var("HF2Q_BENCH_TOKENS")
            .ok()
            .and_then(|v| v.parse().ok())
            .unwrap_or(128);
        // HF2Q_BENCH_N = number of concurrent streams (default 4, max 8) — lets
        // us measure single-stream (N=1) vs batched scaling. For N>4 set
        // HF2Q_SPEC_DECODE_MAX_BATCHED_SLOTS=N to clear the A4 threshold gate.
        let n_streams: usize = std::env::var("HF2Q_BENCH_N")
            .ok()
            .and_then(|v| v.parse().ok())
            .unwrap_or(4)
            .clamp(1, 8);
        let base = [
            vec![1u32, 2, 3],
            vec![4, 5, 6, 7],
            vec![8, 9],
            vec![10, 11, 12, 13, 14],
            vec![15, 16, 17],
            vec![18, 19, 20, 21],
            vec![22, 23],
            vec![24, 25, 26, 27, 28],
        ];
        // HF2Q_BENCH_PROMPT_LEN: pad each stream's prompt to N tokens (long-context
        // throughput at realistic 8k/32k-per-slot). Distinct per-stream token band
        // (well within gemma's 262k vocab) so prompts stay non-degenerate.
        let bench_prompt_len: usize = std::env::var("HF2Q_BENCH_PROMPT_LEN")
            .ok()
            .and_then(|v| v.parse().ok())
            .unwrap_or(0);
        let prompts: Vec<Vec<u32>> = (0..n_streams)
            .map(|i| {
                if bench_prompt_len > 0 {
                    let band = 1000u32 + (i as u32) * 30000u32;
                    (0..bench_prompt_len)
                        .map(|t| band + (t as u32 % 25000))
                        .collect()
                } else {
                    base[i].clone()
                }
            })
            .collect();
        let params = SamplingParams {
            temperature: 0.0,
            max_tokens: bench_tokens,
            ..Default::default()
        };
        let batched_on = std::env::var("HF2Q_BATCHED_BODY").as_deref() == Ok("1");
        let loaded = LoadedModel::load(&load_opts).expect("load");
        let engine = Engine::spawn_with_mode(
            loaded,
            16,
            None,
            EngineMode::SlotAware {
                max_slots: n_streams as u32,
            },
        )
        .expect("spawn SlotAware");
        // ADR-040 iter-I contention probe: vary the probe's tokio worker count
        // (HF2Q_BENCH_TOKIO_THREADS, default 4) to test whether the ~2.45ms/step
        // worker-loop time is core-contention between the model worker thread and
        // the async runtime.
        let tokio_threads: usize = std::env::var("HF2Q_BENCH_TOKIO_THREADS")
            .ok()
            .and_then(|v| v.parse().ok())
            .unwrap_or(4)
            .max(1);
        let rt = tokio::runtime::Builder::new_multi_thread()
            .worker_threads(tokio_threads)
            .enable_all()
            .build()
            .expect("rt");
        // Warm-up generate (model load / pipeline bake) — not timed.
        let _ = rt.block_on(engine.clone().generate(
            prompts[0].clone(),
            SamplingParams {
                temperature: 0.0,
                max_tokens: 8,
                ..Default::default()
            },
        ));
        // ADR-040 §7.S019 determinism-ladder instrument (documented protocol).
        // HF2Q_BENCH_REPEAT=R → run R back-to-back single-process rounds,
        // print a per-run per-stream output fingerprint, then return early
        // (skips the throughput section). Single-process on purpose: process
        // relaunch reloads the ~20GB model per sample AND the readback drain
        // masks the very races this ladder exists to expose (§7.S019).
        // Default = sequential streams (isolates per-stream determinism);
        // HF2Q_BENCH_CONC=1 → each round runs all HF2Q_BENCH_N streams
        // CONCURRENTLY through the live admission path (slot-aware batched
        // prefill + batched decode) — the N>1 concurrency ladder.
        // HF2Q_BENCH_SETTLE_MS=T → sleep T ms between rounds.
        if let Ok(rv) = std::env::var("HF2Q_BENCH_REPEAT") {
            let repeat: usize = rv.parse().unwrap_or(1);
            let settle_ms: u64 = std::env::var("HF2Q_BENCH_SETTLE_MS")
                .ok()
                .and_then(|v| v.parse().ok())
                .unwrap_or(0);
            let concurrent = std::env::var("HF2Q_BENCH_CONC").as_deref() == Ok("1");
            let fnv1a64 = |text: &str| -> u64 {
                let mut h: u64 = 0xcbf29ce484222325;
                for b in text.as_bytes() {
                    h ^= *b as u64;
                    h = h.wrapping_mul(0x100000001b3);
                }
                h
            };
            for run in 0..repeat {
                if concurrent {
                    let results: Vec<GenerationResult> = rt.block_on(async {
                        let mut handles = Vec::new();
                        for p in prompts.iter().cloned() {
                            let eng = engine.clone();
                            let pr = params.clone();
                            handles.push(tokio::spawn(async move {
                                eng.generate(p, pr).await.expect("generate")
                            }));
                        }
                        let mut out = Vec::new();
                        for h in handles {
                            out.push(h.await.expect("join"));
                        }
                        out
                    });
                    for (si, r) in results.iter().enumerate() {
                        eprintln!(
                            "[REPEAT_FPRINT] run={run} stream={si} conc=1 comp_tokens={} text_bytes={} fnv1a64={:016x}",
                            r.completion_tokens, r.text.len(), fnv1a64(&r.text),
                        );
                    }
                } else {
                    for (si, p) in prompts.iter().cloned().enumerate() {
                        let r = rt
                            .block_on(engine.clone().generate(p, params.clone()))
                            .expect("generate");
                        eprintln!(
                            "[REPEAT_FPRINT] run={run} stream={si} conc=0 comp_tokens={} text_bytes={} fnv1a64={:016x} text={:?}",
                            r.completion_tokens, r.text.len(), fnv1a64(&r.text), r.text,
                        );
                    }
                }
                if settle_ms > 0 {
                    std::thread::sleep(std::time::Duration::from_millis(settle_ms));
                }
            }
            return;
        }
        // ADR-040 iter-G: reset the per-category GPU-busy buckets AFTER warm-up
        // (HF2Q_DECODE_CATSPLIT=1) so the table reflects only the timed decode.
        crate::inference::models::gemma4::batched_body::catsplit::reset();
        // ADR-040 §22 host-phase timing reset (HF2Q_HOST_PHASES=1).
        crate::inference::models::gemma4::batched_body::host_phases::reset();
        // ADR-040 §25 barrier-tracking timing reset (HF2Q_BARRIER_NS=1).
        mlx_native::barrier_ns_reset();
        // ADR-040 §26 rerank profiling reset (HF2Q_RERANK_PROFILE=1).
        crate::inference::models::gemma4::batched_head::rerank_profile_reset();
        // ADR-040 §0.21 decode-gap profiling: snapshot process-global GPU
        // dispatch + sync counters around the timed decode (HF2Q_DISP_PROFILE=1).
        let disp0 = mlx_native::dispatch_count();
        let sync0 = mlx_native::sync_count();
        let gpu0 = mlx_native::gpu_busy_ns(); // ADR-040 §0.21 — needs HF2Q_GPU_BUSY=1
        let t0 = std::time::Instant::now();
        let results: Vec<GenerationResult> = rt.block_on(async {
            let mut handles = Vec::new();
            for p in prompts.iter().cloned() {
                let eng = engine.clone();
                let pr = params.clone();
                handles.push(tokio::spawn(async move {
                    eng.generate(p, pr).await.expect("generate")
                }));
            }
            let mut out = Vec::new();
            for h in handles {
                out.push(h.await.expect("join"));
            }
            out
        });
        let elapsed = t0.elapsed();
        // ADR-040 §7.S019: per-stream output fingerprint for the timed round
        // (HF2Q_DUMP_FPRINT=1) — completion token count + FNV-1a-64 hash of
        // the decoded text (a 1:1 function of the output token-id sequence
        // for a fixed tokenizer). Lets us count DISTINCT output sequences
        // across runs.
        if std::env::var("HF2Q_DUMP_FPRINT").is_ok() {
            for (i, r) in results.iter().enumerate() {
                let mut h: u64 = 0xcbf29ce484222325;
                for b in r.text.as_bytes() {
                    h ^= *b as u64;
                    h = h.wrapping_mul(0x100000001b3);
                }
                eprintln!(
                    "[FPRINT] stream={i} comp_tokens={} text_bytes={} fnv1a64={:016x}",
                    r.completion_tokens,
                    r.text.len(),
                    h,
                );
            }
        }
        let total_tokens: usize = results.iter().map(|r| r.completion_tokens).sum();
        let tok_s = total_tokens as f64 / elapsed.as_secs_f64();
        eprintln!(
            "[THROUGHPUT] batched_body={} N={} concurrent: {} tokens in {:.3}s = {:.1} tok/s aggregate ({:.1}/stream)",
            batched_on, n_streams, total_tokens, elapsed.as_secs_f64(), tok_s,
            tok_s / n_streams as f64,
        );
        // ADR-040 iter-G per-category GPU-busy split (HF2Q_DECODE_CATSPLIT=1).
        // The body session was committed at each category boundary; each bucket
        // holds the summed GPU `GPUEndTime-GPUStartTime` of that category's CBs.
        // Reported as GPU-ms PER EMITTED TOKEN (sum of category ns / total tokens)
        // and as % of the category-sum. NOTE: the split serializes CBs production
        // runs as one pipelined CB, so the SUM here OVERSTATES the real GPU-busy
        // step (no inter-CB overlap) — compare the [THROUGHPUT] line with CATSPLIT
        // on vs off for the perturbation, and read this table as a RELATIVE
        // ranking. lm_head is measured by the [DECODE_CATSPLIT] body/head line
        // under HF2Q_DISP_PROFILE.
        if *crate::inference::models::gemma4::batched_body::catsplit::ENABLED {
            let snap = crate::inference::models::gemma4::batched_body::catsplit::snapshot();
            let denom = total_tokens.max(1) as f64;
            let mut rows: Vec<(&str, u64, u64, u64)> = snap
                .into_iter()
                .filter(|(_, ns, _, disp)| *ns > 0 || *disp > 0)
                .collect();
            let sum_ns: u64 = rows.iter().map(|(_, ns, _, _)| *ns).sum::<u64>().max(1);
            let sum_disp: u64 = rows.iter().map(|(_, _, _, d)| *d).sum::<u64>();
            // steps = tokens-per-stream (the batched decode step count).
            let cs_steps = (total_tokens / n_streams.max(1)).max(1);
            // Rank by dispatch count (the fusion target) — the encode-time lever.
            rows.sort_by(|a, b| b.3.cmp(&a.3));
            eprintln!(
                "[CATSPLIT] N={n_streams} per-category GPU-busy + DISPATCH COUNT, ranked by dispatches/step, over {} emitted tokens; total {:.1} disp/step:",
                total_tokens, sum_disp as f64 / cs_steps as f64,
            );
            eprintln!(
                "[CATSPLIT]   {:<40} {:>10} {:>8} {:>10} {:>10}",
                "category", "ms/token", "% step", "disp/step", "cbs/token"
            );
            for (name, ns, cbs, disp) in &rows {
                eprintln!(
                    "[CATSPLIT]   {:<40} {:>10.4} {:>7.1}% {:>10.1} {:>10.1}",
                    name,
                    *ns as f64 / 1e6 / denom,
                    100.0 * *ns as f64 / sum_ns as f64,
                    *disp as f64 / cs_steps as f64,
                    *cbs as f64 / denom,
                );
            }
            eprintln!(
                "[CATSPLIT]   {:<40} {:>10.4} {:>7.1}% (category-sum; CB-serialized, > real overlapped step)",
                "TOTAL",
                sum_ns as f64 / 1e6 / denom,
                100.0,
            );
        }
        if std::env::var("HF2Q_DISP_PROFILE").as_deref() == Ok("1") {
            let d = mlx_native::dispatch_count().saturating_sub(disp0);
            let s = mlx_native::sync_count().saturating_sub(sync0);
            // total_tokens ≈ n_streams * BENCH_TOKENS; decode steps ≈ BENCH_TOKENS
            // (8 tokens/step). Report per-token and per-decode-step.
            let steps = (total_tokens / n_streams).max(1) as u64;
            eprintln!(
                "[DISP_PROFILE] dispatches={d} syncs={s} | per-token: {:.1} disp, {:.2} sync | per-step(N={n_streams}): {:.0} disp, {:.1} sync | {:.1} us/disp wall",
                d as f64 / total_tokens as f64,
                s as f64 / total_tokens as f64,
                d as f64 / steps as f64,
                s as f64 / steps as f64,
                elapsed.as_micros() as f64 / d as f64,
            );
            // ADR-040 §0.21 DECISIVE TEST — GPU-busy vs wall-clock (needs HF2Q_GPU_BUSY=1).
            let gpu_busy = mlx_native::gpu_busy_ns().saturating_sub(gpu0);
            if gpu_busy > 0 {
                let wall_ns = elapsed.as_nanos() as u64;
                let pct = 100.0 * gpu_busy as f64 / wall_ns as f64;
                eprintln!(
                    "[GPU_BUSY] gpu_busy={:.3}s wall={:.3}s → GPU-busy = {:.1}% of wall-clock | per-step(N={n_streams}): gpu {:.2}ms vs wall {:.2}ms → {}",
                    gpu_busy as f64 / 1e9, elapsed.as_secs_f64(), pct,
                    gpu_busy as f64 / 1e6 / steps as f64,
                    wall_ns as f64 / 1e6 / steps as f64,
                    if pct < 70.0 { "CPU-ENCODE/LAUNCH BOUND (hypothesis CONFIRMED)" } else { "GPU-WORK BOUND (CPU-encode refuted)" },
                );
            }
            // ADR-040 §0.21 decode-CATEGORY split — body (embed+30 layers) vs
            // lm_head (final-norm + lm_head(m=N) + softcap), wall-clock of each
            // call (both block on GPU read-back). Localizes the GPU-work-bound step.
            let body_ns = DECODE_BODY_GPU_NS.load(std::sync::atomic::Ordering::Relaxed);
            let lmh_ns = DECODE_LMHEAD_GPU_NS.load(std::sync::atomic::Ordering::Relaxed);
            if body_ns + lmh_ns > 0 {
                let sum = (body_ns + lmh_ns).max(1);
                eprintln!(
                    "[DECODE_CATSPLIT] body(embed+layers) {:.2}ms/step ({:.0}%) | lm_head {:.2}ms/step ({:.0}%) | sum {:.2}ms/step",
                    body_ns as f64 / 1e6 / steps as f64, 100.0 * body_ns as f64 / sum as f64,
                    lmh_ns as f64 / 1e6 / steps as f64, 100.0 * lmh_ns as f64 / sum as f64,
                    sum as f64 / 1e6 / steps as f64,
                );
            }
            // ADR-040 §22 — host-phase breakdown of the wall-vs-GPU-busy gap
            // (HF2Q_HOST_PHASES=1). Shows where the ~8.3ms/step GPU-idle goes.
            let hp = crate::inference::models::gemma4::batched_body::host_phases::snapshot();
            if hp.iter().any(|(_, ns)| *ns > 0) {
                // The last two rows (decode_batch_TOTAL, worker_iter_TOTAL) are
                // reference totals, NOT leaf phases — exclude from the % denom.
                let leaf = hp.len().saturating_sub(2);
                let total: u64 = hp.iter().take(leaf).map(|(_, ns)| *ns).sum();
                eprintln!("[HOST_PHASES] (HF2Q_HOST_PHASES) per-step host wall, {steps} steps:");
                for (i, (name, ns)) in hp.iter().enumerate() {
                    let tag = if i >= leaf { " [ref]" } else { "" };
                    eprintln!(
                        "[HOST_PHASES]   {:<32} {:6.3} ms/step ({:4.1}%){}",
                        name,
                        *ns as f64 / 1e6 / steps as f64,
                        100.0 * *ns as f64 / total.max(1) as f64,
                        tag,
                    );
                }
                eprintln!(
                    "[HOST_PHASES]   {:<32} {:6.3} ms/step (sum of leaf phases)",
                    "TOTAL",
                    total as f64 / 1e6 / steps as f64
                );
                // §25: how much of the serial encode is barrier conflict-tracking.
                let bns = mlx_native::barrier_ns();
                if bns > 0 {
                    eprintln!("[HOST_PHASES]   {:<32} {:6.3} ms/step (barrier_between conflict-tracking; HF2Q_BARRIER_NS)",
                        "of which barrier-track", bns as f64 / 1e6 / steps as f64);
                }
                // §26: split finalize's cost — rerank F64 dots (Metal-no-F64, stuck
                // on host) vs the rest (argmax+candidate-scan, GPU-movable F32).
                let (rr_ns, rr_cand, rr_calls) =
                    crate::inference::models::gemma4::batched_head::rerank_profile();
                if rr_calls > 0 {
                    eprintln!("[HOST_PHASES]   {:<32} {:6.3} ms/step | {:.1} candidates/slot avg ({} calls); HF2Q_RERANK_PROFILE",
                        "of which rerank-F64-dots", rr_ns as f64 / 1e6 / steps as f64,
                        rr_cand as f64 / rr_calls as f64, rr_calls);
                }
            }
        }
        rt.block_on(engine.shutdown()).expect("shutdown");
    }

    /// F1 AC3 — staggered max_tokens: short slots finish early and are
    /// evicted without perturbing peers; each slot's output still matches
    /// its serial reference at its own max_tokens. Exercises mid-batch
    /// eviction + slot refill.
    #[test]
    fn slot_aware_staggered_eviction_no_peer_perturbation() {
        if byte_equiv_skip_unless_gated("slot_aware_staggered_eviction_no_peer_perturbation") {
            return;
        }
        let gguf_path: PathBuf = std::env::var(BYTE_EQUIV_E2E_GGUF_ENV)
            .map(PathBuf::from)
            .expect("HF2Q_BYTE_EQUIV_E2E_GGUF set");
        assert!(gguf_path.exists(), "GGUF missing: {}", gguf_path.display());
        let load_opts = LoadOptions {
            model_path: gguf_path.clone(),
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };

        // Same prompt, divergent max_tokens (5 / 50 / 200) so slots finish
        // at staggered ticks; a 5th prompt is admitted to reuse a freed slot.
        let prompt: Vec<u32> = vec![2, 4, 6, 8];
        let budgets = [5usize, 50, 200];

        // Serial slot-aware reference per (prompt, max_tokens) — SAME
        // forward path as the SlotAware loop (AC4-correct bar).
        let serial_refs: Vec<GenerationResult> = budgets
            .iter()
            .map(|&mt| {
                let pr = SamplingParams {
                    temperature: 0.0,
                    max_tokens: mt,
                    ..Default::default()
                };
                gemma4_serial_slot_aware_ref(&load_opts, &prompt, &pr)
            })
            .collect();
        let rt = tokio::runtime::Builder::new_multi_thread()
            .worker_threads(4)
            .enable_all()
            .build()
            .expect("rt");

        let loaded_slot = LoadedModel::load(&load_opts).expect("load slot");
        let engine_slot =
            Engine::spawn_with_mode(loaded_slot, 8, None, EngineMode::SlotAware { max_slots: 4 })
                .expect("spawn SlotAware{max_slots:4}");
        let slot_results: Vec<GenerationResult> = rt.block_on(async {
            let mut handles = Vec::new();
            for &mt in budgets.iter() {
                let eng = engine_slot.clone();
                let p = prompt.clone();
                let pr = SamplingParams {
                    temperature: 0.0,
                    max_tokens: mt,
                    ..Default::default()
                };
                handles.push(tokio::spawn(async move {
                    eng.generate(p, pr).await.expect("slot")
                }));
            }
            // A 5th request that should reuse the slot freed by the
            // max_tokens=5 completion.
            {
                let eng = engine_slot.clone();
                let p = prompt.clone();
                let pr = SamplingParams {
                    temperature: 0.0,
                    max_tokens: 10,
                    ..Default::default()
                };
                handles.push(tokio::spawn(async move {
                    eng.generate(p, pr).await.expect("slot5")
                }));
            }
            let mut out = Vec::new();
            for h in handles {
                out.push(h.await.expect("join"));
            }
            out
        });

        // The three staggered slots each match their serial reference at
        // their own max_tokens (peers were not perturbed by early eviction).
        for (i, &mt) in budgets.iter().enumerate() {
            assert_genresult_byte_equal(
                &slot_results[i],
                &serial_refs[i],
                &format!("ADR-040 F1 AC3 — staggered slot max_tokens={mt} vs serial ref"),
            );
        }
        rt.block_on(engine_slot.shutdown()).expect("shutdown slot");
    }

    /// ADR-040 §0.12 BLOCKING investigation — characterize the legacy
    /// `generate_once` vs slot-aware forward divergence at the LOGIT level
    /// (not just argmax). Runs the SAME prompt's prefill through both paths
    /// and compares the first-token logits vector: max abs diff, mean abs
    /// diff, whether the greedy argmax flips, and (if it flips) the logit
    /// gap at the flip (near-tie ⇒ benign quant noise; large gap ⇒ bug).
    /// Prints the numbers (run with --nocapture). This isolates the
    /// prefill+KV-representation delta (legacy single-seq F16/F32 KV vs the
    /// slot-aware hybrid TQ-quantized V) from any decode-loop differences.
    #[test]
    fn forward_divergence_legacy_vs_slot_aware_logit_delta() {
        if byte_equiv_skip_unless_gated("forward_divergence_legacy_vs_slot_aware_logit_delta") {
            return;
        }
        let gguf_path: PathBuf = std::env::var(BYTE_EQUIV_E2E_GGUF_ENV)
            .map(PathBuf::from)
            .expect("HF2Q_BYTE_EQUIV_E2E_GGUF set");
        assert!(gguf_path.exists(), "GGUF missing: {}", gguf_path.display());
        let load_opts = LoadOptions {
            model_path: gguf_path.clone(),
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };
        let prompt: Vec<u32> = vec![1u32, 2, 3, 4, 5];
        let max_decode = 1usize; // first-token prefill logits only

        // LEGACY prefill logits.
        let mut legacy = LoadedModel::load(&load_opts).expect("load legacy");
        let LoadedModel::Gemma(lg) = &mut legacy else {
            panic!("expected Gemma")
        };
        let _legacy_tok = lg
            .weights
            .forward_prefill(&prompt, max_decode, &mut lg.ctx)
            .expect("legacy forward_prefill");
        let legacy_logits: Vec<f32> = lg.weights.logits_view().expect("legacy logits").to_vec();

        // SLOT-AWARE prefill logits (production-default hybrid regime).
        let mut slot = LoadedModel::load(&load_opts).expect("load slot");
        let LoadedModel::Gemma(sg) = &mut slot else {
            panic!("expected Gemma")
        };
        sg.provision_multi_seq_kv_for_slot_aware(1)
            .expect("provision n_seqs=1");
        let mut kv = sg.multi_seq_kv.take().expect("multi_seq_kv");
        let mut hybrid = sg.multi_seq_kv_hybrid.take();
        let mut dense = sg.multi_seq_kv_dense.take();
        let mut mlx = sg.multi_seq_kv_mlx.take();
        let _slot_tok = sg
            .weights
            .forward_prefill_with_soft_tokens_slot_aware(
                &prompt,
                &[],
                max_decode,
                &mut sg.ctx,
                SlotId(0),
                &mut kv,
                hybrid.as_mut(),
                dense.as_mut(),
                mlx.as_mut(),
            )
            .expect("slot-aware prefill");
        let slot_logits: Vec<f32> = sg.weights.logits_view().expect("slot logits").to_vec();
        sg.multi_seq_kv = Some(kv);
        sg.multi_seq_kv_hybrid = hybrid;
        sg.multi_seq_kv_dense = dense;
        sg.multi_seq_kv_mlx = mlx;

        assert_eq!(
            legacy_logits.len(),
            slot_logits.len(),
            "logit vocab size mismatch — structural, not a numeric delta"
        );

        // Quantify the delta.
        let mut max_abs = 0.0f32;
        let mut sum_abs = 0.0f64;
        for (a, b) in legacy_logits.iter().zip(slot_logits.iter()) {
            let d = (a - b).abs();
            if d > max_abs {
                max_abs = d;
            }
            sum_abs += d as f64;
        }
        let mean_abs = sum_abs / legacy_logits.len() as f64;
        let argmax = |v: &[f32]| -> usize {
            v.iter()
                .enumerate()
                .max_by(|(_, x), (_, y)| x.partial_cmp(y).unwrap())
                .map(|(i, _)| i)
                .unwrap()
        };
        let legacy_arg = argmax(&legacy_logits);
        let slot_arg = argmax(&slot_logits);
        let flipped = legacy_arg != slot_arg;
        // Logit gap at the legacy argmax: how close was the runner-up? A
        // small gap means the argmax sits on a near-tie (a tiny perturbation
        // flips it — benign). A large gap that still flips ⇒ structural bug.
        let mut legacy_top2 = legacy_logits.clone();
        legacy_top2.sort_by(|a, b| b.partial_cmp(a).unwrap());
        let legacy_gap = legacy_top2[0] - legacy_top2[1];

        eprintln!(
            "[fwd-divergence PREFILL] vocab={} max_abs_logit_diff={:.6} mean_abs_logit_diff={:.6} \
             legacy_argmax={} slot_argmax={} argmax_flipped={} legacy_top1_minus_top2={:.6} \
             max_abs_as_frac_of_gap={:.4}",
            legacy_logits.len(),
            max_abs,
            mean_abs,
            legacy_arg,
            slot_arg,
            flipped,
            legacy_gap,
            if legacy_gap > 0.0 {
                max_abs / legacy_gap
            } else {
                f32::INFINITY
            },
        );
        assert!(
            legacy_logits.iter().all(|x| x.is_finite()),
            "legacy logits non-finite"
        );
        assert!(
            slot_logits.iter().all(|x| x.is_finite()),
            "slot logits non-finite"
        );

        // ── DECODE-STEP comparison ───────────────────────────────────────
        // The prefill logits matched exactly above ⇒ any divergence emerges
        // during DECODE (KV readback). Drive BOTH paths greedily in LOCKSTEP
        // feeding the SAME token each step (legacy's argmax) so we compare
        // logits at the SAME position with the SAME input — isolating the
        // forward/KV-readback delta from input drift. Report, per decode
        // position: max abs logit diff, whether argmax agrees, and the
        // legacy top1-top2 gap (to judge near-tie).
        let mut legacy_re = LoadedModel::load(&load_opts).expect("reload legacy");
        let LoadedModel::Gemma(lg2) = &mut legacy_re else {
            panic!("expected Gemma")
        };
        let mut slot_re = LoadedModel::load(&load_opts).expect("reload slot");
        let LoadedModel::Gemma(sg2) = &mut slot_re else {
            panic!("expected Gemma")
        };
        sg2.provision_multi_seq_kv_for_slot_aware(1)
            .expect("provision n_seqs=1");
        let mut kv2 = sg2.multi_seq_kv.take().expect("kv2");
        let mut hyb2 = sg2.multi_seq_kv_hybrid.take();
        let mut den2 = sg2.multi_seq_kv_dense.take();
        let mut mlx2 = sg2.multi_seq_kv_mlx.take();
        let n_decode = 12usize;
        let l_first = lg2
            .weights
            .forward_prefill(&prompt, n_decode, &mut lg2.ctx)
            .expect("lp");
        let s_first = sg2
            .weights
            .forward_prefill_with_soft_tokens_slot_aware(
                &prompt,
                &[],
                n_decode,
                &mut sg2.ctx,
                SlotId(0),
                &mut kv2,
                hyb2.as_mut(),
                den2.as_mut(),
                mlx2.as_mut(),
            )
            .expect("sp");
        assert_eq!(
            l_first, s_first,
            "first token already differs (contradicts prefill match)"
        );
        let mut feed = l_first;
        let mut first_argmax_flip: Option<usize> = None;
        for step in 1..n_decode {
            let pos = prompt.len() + step - 1;
            let mut p: Option<crate::inference::models::gemma4::profile::TokenProfile> = None;
            let l_tok = lg2
                .weights
                .forward_decode(feed, pos, &mut lg2.ctx, &mut p)
                .expect("ld");
            let l_log: Vec<f32> = lg2.weights.logits_view().expect("ll").to_vec();
            let mut p2: Option<crate::inference::models::gemma4::profile::TokenProfile> = None;
            let s_tok = sg2
                .weights
                .forward_decode_slot_aware(
                    feed,
                    pos,
                    &mut sg2.ctx,
                    &mut p2,
                    SlotId(0),
                    &mut kv2,
                    hyb2.as_mut(),
                    den2.as_mut(),
                    mlx2.as_mut(),
                )
                .expect("sd");
            let s_log: Vec<f32> = sg2.weights.logits_view().expect("sl").to_vec();
            let mut mx = 0.0f32;
            for (a, b) in l_log.iter().zip(s_log.iter()) {
                let d = (a - b).abs();
                if d > mx {
                    mx = d;
                }
            }
            let mut t2 = l_log.clone();
            t2.sort_by(|a, b| b.partial_cmp(a).unwrap());
            let gap = t2[0] - t2[1];
            let flip = l_tok != s_tok;
            if flip && first_argmax_flip.is_none() {
                first_argmax_flip = Some(step);
            }
            eprintln!(
                "[fwd-divergence DECODE step={step} pos={pos}] max_abs_logit_diff={mx:.6} \
                 legacy_tok={l_tok} slot_tok={s_tok} flip={flip} legacy_gap={gap:.6} \
                 frac_of_gap={:.4}",
                if gap > 0.0 { mx / gap } else { f32::INFINITY }
            );
            feed = l_tok; // lockstep on legacy's stream
        }
        eprintln!(
            "[fwd-divergence SUMMARY] prefill_logits_identical={} first_decode_argmax_flip_step={:?}",
            max_abs == 0.0, first_argmax_flip
        );
        // LOAD-BEARING PIN (ADR-040 §0.12 verdict): the slot-aware forward
        // is NUMERICALLY IDENTICAL to the legacy NON-batched forward at
        // n_seqs=1 — same prefill logits, same per-step decode argmax. The
        // legacy-vs-slot-aware divergence seen end-to-end is NOT a forward
        // bug; it is the batched-vs-non-batched PREFILL delta (generate_once
        // defaults to forward_prefill_batched). If this assertion ever
        // fails, the slot-aware forward has genuinely diverged from the
        // model's reference math — a real bug, stop and investigate.
        assert!(
            max_abs == 0.0 && first_argmax_flip.is_none(),
            "ADR-040 §0.12: slot-aware forward diverged from the legacy \
             non-batched forward (prefill max_abs={max_abs}, first_flip={:?}) \
             — this is a forward-correctness regression, NOT the benign \
             batched-prefill delta.",
            first_argmax_flip
        );
        sg2.multi_seq_kv = Some(kv2);
        sg2.multi_seq_kv_hybrid = hyb2;
        sg2.multi_seq_kv_dense = den2;
        sg2.multi_seq_kv_mlx = mlx2;

        // ── FULL-GENERATE wrapper comparison ─────────────────────────────
        // The forwards are identical (above). So if generate_once (legacy
        // full fn) differs from the slot-aware full fn, the divergence is in
        // the GENERATE-LOOP WRAPPER (greedy fast-path token capture, prompt-
        // cache, sampling, first-token handling), NOT the forward.
        let params = SamplingParams {
            temperature: 0.0,
            max_tokens: 16,
            ..Default::default()
        };
        let mut legacy_gen = LoadedModel::load(&load_opts).expect("load legacy gen");
        let LoadedModel::Gemma(lg3) = &mut legacy_gen else {
            panic!("expected Gemma")
        };
        let r_legacy = generate_once(lg3, &prompt, &params, None).expect("generate_once");
        let r_slot_ref = gemma4_serial_slot_aware_ref(&load_opts, &prompt, &params);
        eprintln!(
            "[fwd-divergence WRAPPER] generate_once vs serial_slot_aware: text_match={} \
             legacy_completion_tokens={} slot_completion_tokens={} legacy_finish={} slot_finish={}",
            r_legacy.text == r_slot_ref.text,
            r_legacy.completion_tokens,
            r_slot_ref.completion_tokens,
            r_legacy.finish_reason,
            r_slot_ref.finish_reason,
        );
        if r_legacy.text != r_slot_ref.text {
            eprintln!("[fwd-divergence WRAPPER] legacy_text={:?}", r_legacy.text);
            eprintln!("[fwd-divergence WRAPPER]   slot_text={:?}", r_slot_ref.text);
        }
    }

    // ---------------------------------------------------------------------------
    // ADR-040 Phase C iter-2a (C2b) — H2 test: two sequential requests
    // through the SerialFifo path must produce byte-identical results
    // pair-for-pair (no inter-request state leak introduced by the
    // admit→drive→release wrap landed at iter-2a in this commit).
    //
    // Source: dossier `docs/research/adr040-c2-wiring-dossier-2026-05-24.md`
    // §2.11 H2 + §3 hypothesis matrix + §4 iter-2a step 7.
    //
    // cfa-iter-C2.5 M3 rewrite (this iter): the pre-rewrite H2 compared
    // the SerialFifo engine to itself (two requests through ONE engine
    // built via `spawn_with_mode(SerialFifo)`) which proved
    // run-to-run determinism but NOT pre-vs-post-C2 byte-equivalence —
    // a regression that mutated BOTH requests in the same way would
    // sail through. The rewrite uses TWO engines (pre-C2 3-arg
    // `Engine::spawn` vs iter-1.5 `spawn_with_mode(SerialFifo)`) AND
    // distinct sequential prompts (p1 then p2) so:
    //   - pairwise byte-equality (engine_a r1 vs engine_b r1; engine_a
    //     r2 vs engine_b r2) catches any inter-request state leak that
    //     manifests only on the WRAPPED path.
    //   - distinct prompts catch state leak that's only visible across
    //     a prompt change (a stale KV from r1 corrupting r2's prefill).
    //   - the additional same-prompt-twice guard (a_r1 vs a_r1_again on
    //     engine_a, b_r1 vs b_r1_again on engine_b) pins intra-engine
    //     determinism so a state-leak that would have masked a true
    //     positive on the cross-engine compare is itself flagged.
    //   - the `assert_ne!(a_r1, a_r2)` vacuous-test guard rejects a
    //     fixture where the two distinct prompts produced the same
    //     output (would render the inter-request leak assertions
    //     trivially true).
    //
    // Falsifies what claim: "Under EngineMode::SerialFifo, the
    // `worker_run` admit→drive→release wrap landed at C2b does NOT
    // introduce inter-request state leakage that is observable as a
    // byte-divergence against the pre-C2 3-arg `Engine::spawn` path
    // running the same prompt sequence."
    //
    // Cost-to-falsify: 2 days per dossier H2 row (additional Qwen35
    // persistent_kv_cache lifecycle correctness exercised). Iter-C2.5
    // ships the rewrite in skip mode (same env gate as H1); the live
    // E2E mode requires HF2Q_BYTE_EQUIV_E2E=1 +
    // HF2Q_BYTE_EQUIV_E2E_GGUF=<path>.
    //
    // Vacuous-test guards (M3 strengthened):
    //   1. result must have non-empty text OR completion_tokens > 0
    //      (silent fixture cannot pass trivially).
    //   2. distinct prompts must produce distinct outputs on engine_a
    //      (rejects a fixture where the two prompts happen to map to
    //      the same model output — would make the sequence-leak
    //      assertion vacuous).
    //
    // Stakes if FALSIFIES: the persistent_kv_cache lifecycle is
    // incorrect — likely a missed scheduler.release / drop_seq between
    // requests in the FifoSerial arm OR the worker_run wrap leaks
    // state via shared `loaded`. The fix is localized.
    // ---------------------------------------------------------------------------
    #[test]
    fn engine_serial_fifo_two_sequential_requests_no_state_leak() {
        if byte_equiv_skip_unless_gated("engine_serial_fifo_two_sequential_requests_no_state_leak")
        {
            return;
        }

        let gguf_path: PathBuf = std::env::var(BYTE_EQUIV_E2E_GGUF_ENV)
            .map(PathBuf::from)
            .unwrap_or_else(|_| {
                panic!(
                    "ADR-040 C2b H2: {BYTE_EQUIV_E2E_ENV_GATE}=1 set without \
                     {BYTE_EQUIV_E2E_GGUF_ENV}=<path>. The H2 sequential pin \
                     needs a real GGUF on disk to drive two `generate` calls \
                     through the SerialFifo worker."
                )
            });
        assert!(
            gguf_path.exists(),
            "ADR-040 C2b H2: {BYTE_EQUIV_E2E_GGUF_ENV} points to a missing \
             file: {}",
            gguf_path.display()
        );

        // cfa-iter-C2.5 M3: build TWO independent LoadedModel instances
        // from the SAME GGUF byte source — engine_a via the pre-C2
        // 3-arg `Engine::spawn` entry point and engine_b via the
        // iter-1.5 `spawn_with_mode(SerialFifo)` entry point. (Mirrors
        // H1's two-engine construction.) The same SamplingParams flow
        // through both; only the WRAPPING path differs.
        let load_opts = LoadOptions {
            model_path: gguf_path.clone(),
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };
        let loaded_a = LoadedModel::load(&load_opts).expect("LoadedModel::load (a, H2)");
        let loaded_b = LoadedModel::load(&load_opts).expect("LoadedModel::load (b, H2)");
        let queue_capacity: usize = 4;
        let kv_cache_budget_bytes: Option<u64> = None;

        let engine_a = Engine::spawn(loaded_a, queue_capacity, kv_cache_budget_bytes);
        let engine_b = Engine::spawn_with_mode(
            loaded_b,
            queue_capacity,
            kv_cache_budget_bytes,
            EngineMode::SerialFifo,
        )
        .expect(
            "ADR-040 iter-1.5 F1: EngineMode::SerialFifo MUST succeed at \
             spawn_with_mode (it delegates to 3-arg spawn)",
        );

        let params = SamplingParams {
            temperature: 0.0,
            max_tokens: 16,
            ..Default::default()
        };

        // cfa-iter-C2.5 M3: DISTINCT prompts. p1 and p2 cover different
        // token regions so any state leaked from r1 into r2's prefill
        // would observably perturb r2's logits / sampled tokens.
        let prompt_1: Vec<u32> = vec![1u32, 2, 3, 4, 5];
        let prompt_2: Vec<u32> = vec![6u32, 7, 8, 9, 10];

        let rt = tokio::runtime::Builder::new_current_thread()
            .enable_all()
            .build()
            .expect("build current-thread tokio runtime");

        // Sequence A: p1 then p2 through pre-C2 path.
        let a_r1 = rt
            .block_on(engine_a.generate(prompt_1.clone(), params.clone()))
            .expect("engine_a generate request 1 (H2 M3)");
        let a_r2 = rt
            .block_on(engine_a.generate(prompt_2.clone(), params.clone()))
            .expect("engine_a generate request 2 (H2 M3)");
        // Sequence B: same p1 then p2 sequence through C2b-wrapped path.
        let b_r1 = rt
            .block_on(engine_b.generate(prompt_1.clone(), params.clone()))
            .expect("engine_b generate request 1 (H2 M3)");
        let b_r2 = rt
            .block_on(engine_b.generate(prompt_2.clone(), params.clone()))
            .expect("engine_b generate request 2 (H2 M3)");

        // Vacuous-test guard #1: a_r1 must have produced something.
        assert!(
            !a_r1.text.is_empty() || a_r1.completion_tokens > 0,
            "ADR-040 C2b H2 vacuous test #1: a_r1 produced empty text AND \
             zero completion_tokens (text={:?}, completion_tokens={}) — \
             use a non-trivial prompt or a fixture with deterministic \
             non-empty output.",
            a_r1.text,
            a_r1.completion_tokens,
        );

        // Vacuous-test guard #2 (M3 NEW): distinct prompts MUST produce
        // distinct outputs on engine_a. Without this, the
        // `assert_eq!(a_r2.text, b_r2.text)` sequence-leak assertion
        // collapses to the same shape as the request-1 assertion.
        assert_ne!(
            a_r1.text, a_r2.text,
            "ADR-040 C2b H2 M3 vacuous test #2: distinct prompts p1 + p2 \
             produced IDENTICAL outputs on engine_a — the sequence-leak \
             assertion below would be vacuous. Pick prompts that diverge \
             under the greedy decoder. (a_r1.text == a_r2.text == {:?}.)",
            a_r1.text
        );

        // PAIRWISE byte-equality at request 1: pre-C2 vs C2-wrapped.
        // (Same shape as H1; pins that the wrapping is byte-identical
        // on the FIRST request through both engines.)
        assert_eq!(
            a_r1.text, b_r1.text,
            "ADR-040 C2b H2 M3 FALSIFIED at r1: spawn_with_mode(SerialFifo) \
             `text` differs from 3-arg spawn for request 1 (prompt_1)."
        );
        assert_eq!(
            a_r1.reasoning_text, b_r1.reasoning_text,
            "ADR-040 C2b H2 M3 FALSIFIED at r1: `reasoning_text` differs."
        );
        assert_eq!(
            a_r1.prompt_tokens, b_r1.prompt_tokens,
            "ADR-040 C2b H2 M3 FALSIFIED at r1: `prompt_tokens` differs."
        );
        assert_eq!(
            a_r1.completion_tokens, b_r1.completion_tokens,
            "ADR-040 C2b H2 M3 FALSIFIED at r1: `completion_tokens` differs."
        );
        assert_eq!(
            a_r1.reasoning_tokens, b_r1.reasoning_tokens,
            "ADR-040 C2b H2 M3 FALSIFIED at r1: `reasoning_tokens` differs."
        );
        assert_eq!(
            a_r1.cached_tokens, b_r1.cached_tokens,
            "ADR-040 C2b H2 M3 FALSIFIED at r1: `cached_tokens` differs."
        );
        assert_eq!(
            a_r1.finish_reason, b_r1.finish_reason,
            "ADR-040 C2b H2 M3 FALSIFIED at r1: `finish_reason` differs."
        );
        assert_eq!(
            a_r1.logprobs, b_r1.logprobs,
            "ADR-040 C2b H2 M3 FALSIFIED at r1: `logprobs` differs."
        );

        // PAIRWISE byte-equality at request 2 — the LOAD-BEARING
        // sequence-leak pin. If the C2b wrap leaks state between r1
        // and r2, b_r2 would differ from a_r2 EVEN THOUGH b_r1 matched
        // a_r1 (the leak only manifests on the second request).
        assert_eq!(
            a_r2.text, b_r2.text,
            "ADR-040 C2b H2 M3 FALSIFIED at r2 (SEQUENCE LEAK): \
             spawn_with_mode(SerialFifo) `text` differs from 3-arg spawn \
             for request 2 (prompt_2). State leaked between r1 and r2 in \
             the wrapped path that is not present in pre-C2."
        );
        assert_eq!(
            a_r2.reasoning_text, b_r2.reasoning_text,
            "ADR-040 C2b H2 M3 FALSIFIED at r2: `reasoning_text` differs."
        );
        assert_eq!(
            a_r2.prompt_tokens, b_r2.prompt_tokens,
            "ADR-040 C2b H2 M3 FALSIFIED at r2: `prompt_tokens` differs."
        );
        assert_eq!(
            a_r2.completion_tokens, b_r2.completion_tokens,
            "ADR-040 C2b H2 M3 FALSIFIED at r2: `completion_tokens` differs."
        );
        assert_eq!(
            a_r2.reasoning_tokens, b_r2.reasoning_tokens,
            "ADR-040 C2b H2 M3 FALSIFIED at r2: `reasoning_tokens` differs."
        );
        assert_eq!(
            a_r2.cached_tokens, b_r2.cached_tokens,
            "ADR-040 C2b H2 M3 FALSIFIED at r2: `cached_tokens` differs."
        );
        assert_eq!(
            a_r2.finish_reason, b_r2.finish_reason,
            "ADR-040 C2b H2 M3 FALSIFIED at r2: `finish_reason` differs."
        );
        assert_eq!(
            a_r2.logprobs, b_r2.logprobs,
            "ADR-040 C2b H2 M3 FALSIFIED at r2: `logprobs` differs."
        );

        // cfa-iter-C2.5 M3 ADDITIONAL: same-prompt-twice guard. After
        // the distinct-prompt sequence, replaying prompt_1 on both
        // engines MUST produce the same result as the first time it
        // was issued (no state leaked from the intervening prompt_2
        // request, no scheduler counter drift that affected sampling).
        let a_r1_again = rt
            .block_on(engine_a.generate(prompt_1.clone(), params.clone()))
            .expect("engine_a re-issue prompt_1 (H2 M3 intra-engine determinism)");
        let b_r1_again = rt
            .block_on(engine_b.generate(prompt_1, params))
            .expect("engine_b re-issue prompt_1 (H2 M3 intra-engine determinism)");
        assert_eq!(
            a_r1.text, a_r1_again.text,
            "ADR-040 C2b H2 M3 FALSIFIED (intra-engine_a leak): re-issuing \
             prompt_1 on engine_a after the p1→p2 sequence produced different \
             text — the pre-C2 path itself shows intra-request state leakage."
        );
        assert_eq!(
            b_r1.text, b_r1_again.text,
            "ADR-040 C2b H2 M3 FALSIFIED (intra-engine_b leak): re-issuing \
             prompt_1 on engine_b after the p1→p2 sequence produced different \
             text — the C2b-wrapped path leaks state across requests."
        );

        // ADR-040 C2b additional H2 surface: after all requests
        // complete, both engines' scheduler stats reflect their admit/
        // release pairs.
        let stats_a = engine_a.scheduler_stats();
        let stats_b = engine_b.scheduler_stats();
        assert_eq!(
            stats_a.policy,
            SchedulerPolicy::FifoSerial,
            "ADR-040 C2b H2 M3: engine_a scheduler_stats policy must report \
             FifoSerial (3-arg spawn defaults to SerialFifo)."
        );
        assert_eq!(
            stats_b.policy,
            SchedulerPolicy::FifoSerial,
            "ADR-040 C2b H2 M3: engine_b scheduler_stats policy must report \
             FifoSerial under explicit SerialFifo mode."
        );
        assert!(
            stats_a.admitted_total >= 3,
            "ADR-040 C2b H2 M3: engine_a admitted_total reflects 3 Generate \
             requests (p1, p2, p1-again); got {}.",
            stats_a.admitted_total
        );
        assert!(
            stats_b.admitted_total >= 3,
            "ADR-040 C2b H2 M3: engine_b admitted_total reflects 3 Generate \
             requests; got {}.",
            stats_b.admitted_total
        );
        assert!(
            stats_a.completed_total >= 3,
            "ADR-040 C2b H2 M3: engine_a completed_total reflects 3 releases; \
             got {}.",
            stats_a.completed_total
        );
        assert!(
            stats_b.completed_total >= 3,
            "ADR-040 C2b H2 M3: engine_b completed_total reflects 3 releases; \
             got {}.",
            stats_b.completed_total
        );
        assert_eq!(
            stats_a.in_flight_slots, 0,
            "ADR-040 C2b H2 M3: engine_a in_flight_slots must be 0 after all \
             requests released."
        );
        assert_eq!(
            stats_b.in_flight_slots, 0,
            "ADR-040 C2b H2 M3: engine_b in_flight_slots must be 0 after all \
             requests released."
        );

        rt.block_on(engine_a.shutdown()).expect("engine_a shutdown");
        rt.block_on(engine_b.shutdown()).expect("engine_b shutdown");
    }

    // ---------------------------------------------------------------------------
    // cfa-iter-C2.5 M2 Approach C — synthetic-fixture engine_scheduler
    // admit→release consistency pin.
    //
    // Approach A (default-on deterministic fixture lifting H1+H2 out
    // of env-gating) is NOT feasible at this iter: the synthetic worker
    // at `make_synthetic_engine_for_test` does not call `worker_run`
    // (it just drains the channel + handles Shutdown), so it has no
    // scheduler to bookkeep. Lifting H1/H2 default-on would require
    // either (i) loading a real GGUF on every CI run (memory + GPU
    // cost that violates "do not oom us"), or (ii) refactoring the
    // synthetic worker to run `worker_run` against a fake LoadedModel
    // (invasive — touches production code paths via the LoadedModel
    // enum). Approach B (operator-run gate documentation in §6.1.10)
    // is the chosen mitigation for H1+H2; this Approach C test pins
    // an ORTHOGONAL property: that constructing engines via the two
    // public spawn entry points (`spawn` + `spawn_with_mode(SerialFifo)`)
    // produces a `SchedulerStats` snapshot with the same shape (policy,
    // queue_capacity, and zero counters at construction time).
    //
    // The test does NOT exercise the worker thread's
    // admit/advance/release wiring — the synthetic worker drops
    // Generate requests silently. But it DOES exercise the snapshot
    // initialization at `Engine::spawn` + `Engine::spawn_with_mode`,
    // which is the surface that a future refactor of the snapshot
    // shape would touch. Catches: a future regression where the two
    // spawn entry points seed `scheduler_stats_snapshot` differently
    // (e.g. different `queue_capacity`, different policy, non-zero
    // initial counters).
    // ---------------------------------------------------------------------------
    #[test]
    fn engine_scheduler_admit_release_consistency_under_synthetic_fixture() {
        // Build a synthetic Gemma engine via the 3-arg `Engine::spawn`-
        // shaped construction (no real load — the helper hand-rolls the
        // EngineInner with a no-op worker). Then build a second
        // synthetic engine via the same helper and assert their
        // scheduler_stats snapshots are SHAPE-equivalent.
        let engine_a = make_synthetic_engine_for_test(LoadedArch::Gemma);
        let engine_b = make_synthetic_engine_for_test(LoadedArch::Gemma);

        let stats_a = engine_a.scheduler_stats();
        let stats_b = engine_b.scheduler_stats();

        assert_eq!(
            stats_a.policy,
            SchedulerPolicy::FifoSerial,
            "cfa-iter-C2.5 M2 C: synthetic engine_a scheduler_stats policy \
             must be FifoSerial (matches `make_synthetic_engine_for_test` \
             default mode at engine.rs:892)."
        );
        assert_eq!(
            stats_b.policy, stats_a.policy,
            "cfa-iter-C2.5 M2 C: synthetic engine_b policy must match \
             engine_a — both helpers must initialize the snapshot identically."
        );
        assert_eq!(
            stats_a.queue_capacity, stats_b.queue_capacity,
            "cfa-iter-C2.5 M2 C: synthetic engines must seed identical \
             queue_capacity (both helpers use `8` per engine.rs:851 + 899)."
        );
        assert_eq!(
            stats_a.admitted_total, 0,
            "cfa-iter-C2.5 M2 C: freshly-constructed synthetic engine_a \
             admitted_total must start at 0 (no admit yet)."
        );
        assert_eq!(
            stats_a.completed_total, 0,
            "cfa-iter-C2.5 M2 C: freshly-constructed synthetic engine_a \
             completed_total must start at 0."
        );
        assert_eq!(
            stats_a.in_flight_slots, 0,
            "cfa-iter-C2.5 M2 C: freshly-constructed synthetic engine_a \
             in_flight_slots must start at 0."
        );
        assert_eq!(
            stats_a.rejected_429_total, 0,
            "cfa-iter-C2.5 M2 C: freshly-constructed synthetic engine_a \
             rejected_429_total must start at 0."
        );

        // engine_a and engine_b are independent instances — the snapshot
        // mutex is per-engine so mutating one cannot affect the other.
        // (Defensive pin against a future refactor that shares the
        // snapshot via Arc + breaks per-engine isolation.)
        assert_eq!(
            stats_a, stats_b,
            "cfa-iter-C2.5 M2 C: two independent synthetic engines must \
             produce structurally-equivalent SchedulerStats snapshots — any \
             divergence indicates a future refactor that shares mutable \
             state between independently-constructed Engine instances."
        );

        // engine_a + engine_b have independent mode() reports too —
        // pins the iter-1.5 F1 invariant that `Engine::mode()` is
        // honest about the configured mode.
        assert!(
            matches!(engine_a.mode(), EngineMode::SerialFifo),
            "cfa-iter-C2.5 M2 C: synthetic engine_a.mode() must be \
             SerialFifo (mirrors the helper's initial_mode at engine.rs:892)."
        );
        assert!(
            matches!(engine_b.mode(), EngineMode::SerialFifo),
            "cfa-iter-C2.5 M2 C: synthetic engine_b.mode() must be \
             SerialFifo."
        );

        // max_slots is the SchedulerPolicy::FifoSerial cap (always 1
        // per engine.rs:847 / 895), independent of queue_capacity.
        assert_eq!(
            engine_a.max_slots(),
            1,
            "cfa-iter-C2.5 M2 C: synthetic engine_a.max_slots() must be 1 \
             under FifoSerial."
        );
        assert_eq!(
            engine_b.max_slots(),
            1,
            "cfa-iter-C2.5 M2 C: synthetic engine_b.max_slots() must be 1."
        );
    }
}

// ---------------------------------------------------------------------------
// Wave 3 W-A2 — streaming PromptCache replay tests
//
// Drive `replay_cached_streaming_response` directly through a real
// `mpsc::channel`, drain the receiver, and assert SSE event shape.
// Single-shot per test — no full engine, no live model load.  Mirrors the
// same direct-helper pattern wave-2.8 finalize_streaming_tool_state_tests
// used (no sham reconstruction of the production codepath).
// ---------------------------------------------------------------------------
#[cfg(test)]
mod streaming_prompt_cache_replay_tests {
    use super::super::sse::{DeltaKind, GenerationEvent};
    use super::*;

    /// Build a `GenerationResult` that looks like a non-streaming-origin
    /// cache entry (post-reasoning-split text + explicit reasoning_text).
    fn cached_non_streaming(text: &str, reasoning: Option<&str>) -> GenerationResult {
        GenerationResult {
            text: text.to_string(),
            reasoning_text: reasoning.map(|s| s.to_string()),
            prompt_tokens: 7,
            completion_tokens: 5,
            reasoning_tokens: reasoning.map(|_| 3),
            finish_reason: "stop",
            prefill_duration: Duration::ZERO,
            decode_duration: Duration::ZERO,
            cached_tokens: 7,
            logprobs: None,
        }
    }

    /// Drain all events the helper produces synchronously.  Helper writes
    /// to a tokio mpsc via `blocking_send`, which works against a tokio
    /// receiver from a non-async context if the channel has capacity (we
    /// use 32, well over what any single replay needs).
    fn drain(rx: &mut mpsc::Receiver<GenerationEvent>) -> Vec<GenerationEvent> {
        let mut out = Vec::new();
        while let Ok(ev) = rx.try_recv() {
            out.push(ev);
        }
        out
    }

    #[test]
    fn replay_emits_content_then_done_for_plain_text() {
        let (tx, mut rx) = mpsc::channel(32);
        let cached = cached_non_streaming("Hello, world!", None);

        let res = replay_cached_streaming_response(
            &cached,
            None, // no registration ⇒ everything routes as Content
            ToolCallPolicy::Auto,
            &EventSink::new(&tx),
        );
        assert!(res.is_ok(), "replay must succeed when no client disconnect");

        // Drop the sender so try_recv finds events without blocking.
        drop(tx);
        let events = drain(&mut rx);

        // Expected: 1 Delta(Content) + 1 Done.
        assert_eq!(events.len(), 2, "got events: {events:?}");
        match &events[0] {
            GenerationEvent::Delta {
                kind: DeltaKind::Content,
                text,
            } => {
                assert_eq!(text, "Hello, world!");
            }
            other => panic!("expected Delta(Content); got {other:?}"),
        }
        match &events[1] {
            GenerationEvent::Done {
                finish_reason,
                prompt_tokens,
                completion_tokens,
                stats,
            } => {
                assert_eq!(*finish_reason, "stop");
                assert_eq!(*prompt_tokens, 7);
                assert_eq!(*completion_tokens, 5);
                // Cache-hit signal: cached_prompt_tokens populated, timings zeroed.
                assert_eq!(stats.cached_prompt_tokens, Some(7));
                assert_eq!(stats.prefill_time_secs, Some(0.0));
                assert_eq!(stats.decode_time_secs, Some(0.0));
            }
            other => panic!("expected Done; got {other:?}"),
        }
    }

    #[test]
    fn replay_emits_reasoning_then_content_when_reasoning_text_set() {
        let (tx, mut rx) = mpsc::channel(32);
        // Non-streaming-origin entry: reasoning was split out into its own
        // field; the assembled `text` is post-split content only.
        let cached = cached_non_streaming("the answer", Some("let me think..."));

        let res = replay_cached_streaming_response(
            &cached,
            None,
            ToolCallPolicy::Auto,
            &EventSink::new(&tx),
        );
        assert!(res.is_ok());
        drop(tx);
        let events = drain(&mut rx);

        // Expected: Reasoning, Content, Done.
        assert_eq!(events.len(), 3, "got events: {events:?}");
        match &events[0] {
            GenerationEvent::Delta {
                kind: DeltaKind::Reasoning,
                text,
            } => {
                assert_eq!(text, "let me think...");
            }
            other => panic!("expected Delta(Reasoning); got {other:?}"),
        }
        match &events[1] {
            GenerationEvent::Delta {
                kind: DeltaKind::Content,
                text,
            } => {
                assert_eq!(text, "the answer");
            }
            other => panic!("expected Delta(Content); got {other:?}"),
        }
        assert!(matches!(events[2], GenerationEvent::Done { .. }));
    }

    /// Replay a cache entry whose `text` contains tool-call markers
    /// (mirrors a streaming-origin cache entry where `accumulated_text`
    /// captures the raw pre-split stream).  The replay must re-route
    /// through the live-decode tool-call splitter so the SSE shape is
    /// `ToolCallDelta` events, not raw content text.
    #[test]
    fn replay_routes_tool_call_markers_to_tool_call_delta_events() {
        let (tx, mut rx) = mpsc::channel(32);

        // Use the gemma4 registration so we have real tool open/close
        // markers + a body-parser registered.
        let reg = match super::super::registry::find_for("gemma4-27b-it") {
            Some(r) => r,
            None => {
                eprintln!("gemma4 registration absent; skipping tool-call replay test");
                return;
            }
        };
        let (open, close) = match (reg.tool_open, reg.tool_close) {
            (Some(o), Some(c)) => (o, c),
            _ => {
                eprintln!("gemma4 has no tool markers; skipping tool-call replay test");
                return;
            }
        };

        // Construct a cached `text` shaped like Gemma 4's tool-call output:
        //     "preamble<open>{"name":"foo","arguments":{}}<close>postscript"
        // The body uses the per-model parser-friendly shape; since we don't
        // know gemma4's exact body grammar offline, the assertion focuses
        // on event-class-shape (Content + ToolCallDelta + Content) — NOT
        // on whether parse succeeds.  Both Some(parsed) → ToolCallDelta×2
        // and None (under Auto) → Content fallback are valid replay
        // shapes per `emit_streaming_tool_call_close`'s policy matrix.
        let cached_text =
            format!("preamble {open}{{\"name\":\"foo\",\"arguments\":{{}}}}{close} postscript");
        let cached = GenerationResult {
            text: cached_text,
            reasoning_text: None,
            prompt_tokens: 4,
            completion_tokens: 9,
            reasoning_tokens: None,
            finish_reason: "stop",
            prefill_duration: Duration::ZERO,
            decode_duration: Duration::ZERO,
            cached_tokens: 4,
            logprobs: None,
        };

        let res = replay_cached_streaming_response(
            &cached,
            Some(&reg),
            ToolCallPolicy::Auto,
            &EventSink::new(&tx),
        );
        assert!(res.is_ok(), "replay must succeed");
        drop(tx);
        let events = drain(&mut rx);

        // Must contain at least one Delta (preamble) and a terminal Done.
        assert!(
            events.iter().any(|e| matches!(
                e,
                GenerationEvent::Delta { kind: DeltaKind::Content, text } if text.contains("preamble")
            )),
            "preamble must be emitted as Content delta; got {events:?}"
        );
        let done_idx = events
            .iter()
            .position(|e| matches!(e, GenerationEvent::Done { .. }))
            .expect("Done event missing");
        assert_eq!(done_idx, events.len() - 1, "Done must be last event");

        // Extract the Done and assert cached_tokens surfaces.
        if let GenerationEvent::Done {
            stats,
            finish_reason,
            ..
        } = &events[done_idx]
        {
            assert_eq!(stats.cached_prompt_tokens, Some(4));
            // finish_reason: if the splitter drove ToolCallOpen+Close to
            // completion AND parser succeeded, we expect "tool_calls"; if
            // parser failed under Auto the body re-emits as content and
            // saw_tool_call stays false (cached.finish_reason="stop"
            // wins).  Both are valid here — the test asserts the BRANCH
            // wires correctly, not the per-model parser outcome.
            assert!(
                *finish_reason == "tool_calls" || *finish_reason == "stop",
                "finish_reason should be tool_calls or stop; got {finish_reason:?}"
            );
        }
    }

    /// Sanity: the streaming preroll lookup is gated on the same
    /// eligibility predicate as the non-streaming preroll.  An empty cache
    /// with a default-greedy request returns None — the lookup short-
    /// circuits and the live decode runs.  This is the miss path probe.
    #[test]
    fn empty_cache_lookup_returns_none() {
        let cache = PromptCache::new();
        let params = SamplingParams::default();
        let prompt = vec![1u32, 2, 3];
        assert!(
            cache.lookup(&prompt, &params).is_none(),
            "fresh cache must miss on first request"
        );
    }

    /// After `store`, the SAME prompt + params hits and produces a
    /// `GenerationResult` with `cached_tokens == prompt.len()`.  This is
    /// the same contract the non-streaming preroll relies on — the
    /// streaming replay just consumes that result.
    #[test]
    fn store_then_lookup_round_trips_for_replay() {
        let prompt = vec![10u32, 20, 30, 40];
        let params = SamplingParams::default();
        let result = GenerationResult {
            text: "cached body".into(),
            reasoning_text: None,
            prompt_tokens: prompt.len(),
            completion_tokens: 11,
            reasoning_tokens: None,
            finish_reason: "stop",
            prefill_duration: Duration::ZERO,
            decode_duration: Duration::ZERO,
            cached_tokens: 0,
            logprobs: None,
        };
        let mut cache = PromptCache::new();
        cache.store(&prompt, &params, &result);

        let hit = cache
            .lookup(&prompt, &params)
            .expect("must hit after store");
        assert_eq!(hit.text, "cached body");
        assert_eq!(hit.cached_tokens, prompt.len());
        assert_eq!(hit.completion_tokens, 11);
        assert_eq!(hit.finish_reason, "stop");
    }

    /// Replay returns Err(()) when the receiver is dropped mid-replay —
    /// the production callsite then bumps the cancellation counter.  This
    /// exercises the disconnect path which mirrors the live decode's
    /// `events.blocking_send(...).is_err()` checks.
    #[test]
    fn replay_returns_err_when_receiver_dropped() {
        let (tx, rx) = mpsc::channel(1); // tiny buffer
                                         // Drop receiver so all sends fail.
        drop(rx);

        let cached = cached_non_streaming("anything", None);
        let res = replay_cached_streaming_response(
            &cached,
            None,
            ToolCallPolicy::Auto,
            &EventSink::new(&tx),
        );
        assert!(
            res.is_err(),
            "replay must return Err when receiver was dropped"
        );
    }

    // ---------------------------------------------------------------------
    // Wave 3.5 HIGH-2 — audit-driven splitter-drain tests
    //
    // Audit divergence
    // /tmp/cfa-cfa-20260427-adr005-wave3/codex-review-last.txt
    // "W-A2 streaming cache replay" severity HIGH:
    //
    //   "replay_cached_streaming_response feeds cached.text once at
    //    src/serve/api/engine.rs:2977-3012 and immediately emits Done
    //    at src/serve/api/engine.rs:3015-3048. It never calls
    //    ReasoningSplitter::finish() or ToolCallSplitter::finish(),
    //    even though those splitters hold back tail bytes until
    //    finish at src/serve/api/registry.rs:397-463 and
    //    src/serve/api/registry.rs:587-658. Registered plain-text
    //    cache hits can therefore emit empty/truncated content."
    //
    // The two missed-test gaps the audit cited:
    //   1. "No unit test replays short plain content with a registered
    //      model; replay_emits_content_then_done_for_plain_text passes
    //      registration=None ... bypassing both tail-holding splitters."
    //   2. "No replay test asserts final postscript/tail content after
    //      tool-call markers."
    //
    // The tests below close both gaps and would fail on a regression
    // that removes the new finish() drain calls.
    // ---------------------------------------------------------------------

    /// Wave 3.5 HIGH-2 missed-test #1: a registered-model plain-text
    /// replay must drain the splitter tail before Done.
    ///
    /// The Gemma 4 ToolCallSplitter has `tail_cap = max(open_marker.len,
    /// close_marker.len) = max(12, 12) = 12 bytes` (registry.rs:565).
    /// Cached text shorter than `tail_cap` ends up entirely in the
    /// splitter's tail_buf — `feed()` emits zero events, `finish()` is
    /// the only way to recover the bytes.  Pre-Wave-3.5 replay never
    /// called `finish()`, so the entire response was lost.
    #[test]
    fn replay_emits_tail_content_after_splitter_drain() {
        let reg = match super::super::registry::find_for("gemma4-27b-it") {
            Some(r) => r,
            None => {
                eprintln!("gemma4 registration absent; skipping HIGH-2 drain test");
                return;
            }
        };

        let (tx, mut rx) = mpsc::channel(8);
        // Short plain-text cache entry (< Gemma's 12-byte marker
        // tail_cap).  No marker, no reasoning — pure content.  This
        // is the exact "registered plain-text cache hit" shape the
        // audit cited.
        let cached = cached_non_streaming("hi", None);

        let res = replay_cached_streaming_response(
            &cached,
            Some(&reg), // <-- KEY: registration enables splitter (the bug only fires when splitter is built)
            ToolCallPolicy::Auto,
            &EventSink::new(&tx),
        );
        assert!(res.is_ok(), "replay must succeed");
        drop(tx);
        let events = drain(&mut rx);

        // The cached "hi" MUST appear as a Content delta before Done.
        // Pre-Wave-3.5: the splitter's tail_buf swallowed "hi" entirely
        // because feed() held back the last `tail_cap` bytes and
        // finish() was never called → zero Content deltas → silent
        // data loss on the cache hit.
        let content_text: String = events
            .iter()
            .filter_map(|e| match e {
                GenerationEvent::Delta {
                    kind: DeltaKind::Content,
                    text,
                } => Some(text.as_str()),
                _ => None,
            })
            .collect();
        assert_eq!(
            content_text, "hi",
            "registered plain-text cache replay MUST emit the full \
             cached content as Content delta(s) before Done.  \
             Pre-Wave-3.5 the splitter's tail_buf silently swallowed \
             content shorter than tail_cap (12 bytes for Gemma 4) \
             because finish() was never called.  Audit citation: \
             /tmp/cfa-cfa-20260427-adr005-wave3/codex-review-last.txt \
             'W-A2 streaming cache replay' severity HIGH.\n\
             events: {events:?}"
        );

        // Done must follow.
        assert!(
            matches!(events.last(), Some(GenerationEvent::Done { .. })),
            "Done must be the last event"
        );
    }

    /// Wave 3.5 HIGH-2 missed-test #2: a registered-model replay whose
    /// cached text contains a tool-call marker block PLUS trailing
    /// postscript content must emit BOTH the structured tool-call AND
    /// the postscript content.
    ///
    /// Pre-Wave-3.5 the postscript portion shorter than the splitter's
    /// `tail_cap` bytes would be silently dropped, OR a postscript
    /// whose tail looked like a partial open-marker prefix would be
    /// held back forever.
    #[test]
    fn replay_with_registered_model_emits_tool_call_then_postscript() {
        let reg = match super::super::registry::find_for("gemma4-27b-it") {
            Some(r) => r,
            None => {
                eprintln!("gemma4 registration absent; skipping HIGH-2 postscript test");
                return;
            }
        };
        let (open, close) = match (reg.tool_open, reg.tool_close) {
            (Some(o), Some(c)) => (o, c),
            _ => {
                eprintln!("gemma4 has no tool markers; skipping HIGH-2 postscript test");
                return;
            }
        };

        // Cached text: tool-call block + trailing postscript shorter
        // than `tail_cap` (Gemma's max marker length is 12 bytes; a
        // 5-byte postscript "after" sits entirely in the splitter's
        // tail_buf after feed() returns and is only recoverable via
        // finish()).
        let cached_text = format!("{open}call:foo{{x:1}}{close}after");
        let cached = GenerationResult {
            text: cached_text,
            reasoning_text: None,
            prompt_tokens: 4,
            completion_tokens: 9,
            reasoning_tokens: None,
            finish_reason: "stop",
            prefill_duration: Duration::ZERO,
            decode_duration: Duration::ZERO,
            cached_tokens: 4,
            logprobs: None,
        };

        let (tx, mut rx) = mpsc::channel(16);
        let res = replay_cached_streaming_response(
            &cached,
            Some(&reg),
            ToolCallPolicy::Auto,
            &EventSink::new(&tx),
        );
        assert!(res.is_ok(), "replay must succeed");
        drop(tx);
        let events = drain(&mut rx);

        // Concatenate ALL Content deltas — the postscript "after" MUST
        // appear somewhere.  Pre-Wave-3.5 the splitter's finish() was
        // never called and "after" was silently dropped.
        let content_concat: String = events
            .iter()
            .filter_map(|e| match e {
                GenerationEvent::Delta {
                    kind: DeltaKind::Content,
                    text,
                } => Some(text.clone()),
                _ => None,
            })
            .collect();
        assert!(
            content_concat.contains("after"),
            "postscript content 'after' MUST appear in a Content delta \
             after the tool-call block.  Pre-Wave-3.5 the ToolCallSplitter \
             held the postscript in its tail_buf and finish() was never \
             called → silent postscript loss.  Audit citation: \
             /tmp/cfa-cfa-20260427-adr005-wave3/codex-review-last.txt \
             missed-test 'No replay test asserts final postscript/tail \
             content after tool-call markers'.\n\
             content concat: {content_concat:?}\nevents: {events:?}"
        );

        // Wave 3.6 W-4 strengthening (audit gap from
        // /tmp/cfa-cfa-20260427-adr005-wave3.5/codex-review-last.txt):
        //
        // "asserts postscript content and Done, but does not assert
        //  ToolCallDelta or finish_reason=tool_calls for the parsed
        //  marker block."
        //
        // The cached text is `{open}call:foo{{x:1}}{close}after`.
        // The body `call:foo{{x:1}}` is parseable by parse_gemma4_tool_call
        // → parse_tool_call_body returns Some(ParsedToolCall{name:"foo",
        //   args:{"x":1}}).  emit_streaming_tool_call_close then emits
        // two ToolCallDelta events (name chunk + args chunk) and sets
        // saw_tool_call=true → Done gets finish_reason="tool_calls".
        //
        // The splitter chain must re-classify the marker block into
        // structured ToolCallDelta events identical to a fresh decode.
        let tool_call_deltas: Vec<_> = events
            .iter()
            .filter(|e| matches!(e, GenerationEvent::ToolCallDelta { .. }))
            .collect();
        assert!(
            !tool_call_deltas.is_empty(),
            "Wave 3.6 W-4: MUST emit at least one ToolCallDelta for the \
             parsed `call:foo{{x:1}}` body — the splitter chain re-classifies \
             the marker block into structured ToolCall deltas.  \
             events: {events:?}"
        );

        // The FIRST ToolCallDelta MUST carry the function name (name chunk);
        // subsequent deltas carry arguments only.  Wave 3.7 strengthening
        // per Codex audit MED: previously asserted `events.iter().any()`
        // which would have passed if name appeared on a later delta.
        let first_delta = tool_call_deltas
            .first()
            .expect("at least one ToolCallDelta asserted above");
        match first_delta {
            GenerationEvent::ToolCallDelta { name, .. } => {
                assert_eq!(
                    name.as_deref(),
                    Some("foo"),
                    "Wave 3.6 W-4 (Wave 3.7 strengthened): the FIRST ToolCallDelta \
                     MUST carry `name: Some(\"foo\")`. Got: {first_delta:?}"
                );
            }
            _ => unreachable!("filtered to ToolCallDelta above"),
        }

        // The Done event MUST report finish_reason="tool_calls" because
        // saw_tool_call is set by emit_streaming_tool_call_close when parse
        // succeeds (engine.rs:3184: `if saw_tool_call { "tool_calls" } else ...`).
        // Cached text has finish_reason="stop" but the replay overrides it.
        let done_finish_reason = events
            .iter()
            .find_map(|e| match e {
                GenerationEvent::Done { finish_reason, .. } => Some(*finish_reason),
                _ => None,
            })
            .expect("Done event must be present");
        assert_eq!(
            done_finish_reason, "tool_calls",
            "Wave 3.6 W-4: finish_reason MUST be 'tool_calls' (not '{}') when \
             a tool call was extracted from the cached text during replay. \
             The replay overrides cached.finish_reason (='stop') with \
             'tool_calls' when saw_tool_call=true (engine.rs:3184). \
             events: {events:?}",
            done_finish_reason
        );

        // Sanity: Done must terminate the stream.
        assert!(
            matches!(events.last(), Some(GenerationEvent::Done { .. })),
            "Done must be the last event"
        );
    }

    // -----------------------------------------------------------------
    // ADR-005 Phase 4 iter C — `delta.reasoning_content` extractor
    // unit-level closure tests (2026-05-01).
    //
    // The iter B-2 ToolCallSplitter LANDED in iter-219c (commit
    // `94c0dbe`); the parallel iter C reasoning-content extractor was
    // wired alongside it (W66/W67 path: ReasoningSplitter integrated
    // into `generate_stream_once` + `replay_cached_streaming_response`,
    // schema `ChunkDelta.reasoning_content` + `ChatMessage
    // .reasoning_content` populated, SSE encoder routes on
    // `DeltaKind::Reasoning` per Decision #21). The pre-existing
    // `replay_emits_reasoning_then_content_when_reasoning_text_set`
    // test exercises the **non-streaming-origin** cache entry shape
    // (text post-split + `reasoning_text=Some(...)`); these tests
    // close the **streaming-origin** branch (text contains embedded
    // reasoning markers + `reasoning_text=None`) plus the
    // reasoning + tool-call interleaving contract — both currently
    // only exercised via the env-gated live test
    // `tests/openwebui_reasoning.rs::openwebui_reasoning_streaming_scenario_3`,
    // which is not part of the default `cargo test` baseline.
    //
    // Mantra alignment: no env-gating, no model-load, deterministic,
    // sub-millisecond. Locks the iter C contract at every cargo test
    // invocation so a regression in either splitter wiring or the
    // mutual-exclusion of reasoning vs tool_calls (per OpenAI spec)
    // surfaces loud at unit-test time, not at LIVE-test time.
    // -----------------------------------------------------------------

    /// Iter C streaming-origin shape: cache entry's `text` field carries
    /// the **raw pre-split decoded stream** (markers and all) with
    /// `reasoning_text=None`. The replay helper must run the cached text
    /// through a fresh `ReasoningSplitter` and route the marker-bounded
    /// span as `DeltaKind::Reasoning`, the rest as `DeltaKind::Content`.
    ///
    /// Marker pair: Qwen 3.5/3.6 `<think>` / `</think>` (registered
    /// reasoning markers per `registry::QWEN35`).
    #[test]
    fn replay_routes_streaming_origin_reasoning_markers_to_reasoning_deltas() {
        // Resolve a model-id that maps to QWEN35 registration so the
        // splitter has reasoning markers + tool markers both registered.
        let reg = match super::super::registry::find_for("qwen3.6-27b-dwq46") {
            Some(r) => r,
            None => {
                eprintln!("qwen35 registration absent; skipping iter C streaming-origin test");
                return;
            }
        };
        // Sanity: the registration must have reasoning markers, else
        // the test is degenerate.
        assert!(
            reg.has_reasoning(),
            "iter C contract: qwen35 family MUST have reasoning markers \
             registered; got open={:?} close={:?}",
            reg.reasoning_open,
            reg.reasoning_close,
        );

        // Streaming-origin cache shape: `text` carries the pre-split
        // stream verbatim; `reasoning_text=None` because the LIVE
        // splitter routed reasoning fragments into Reasoning deltas at
        // decode time (no separately-tracked string to replay).
        let cached_text = "<think>let me compute 2+2</think>The answer is 4.";
        let cached = GenerationResult {
            text: cached_text.into(),
            reasoning_text: None,
            prompt_tokens: 5,
            completion_tokens: 12,
            reasoning_tokens: None,
            finish_reason: "stop",
            prefill_duration: Duration::ZERO,
            decode_duration: Duration::ZERO,
            cached_tokens: 5,
            logprobs: None,
        };

        let (tx, mut rx) = mpsc::channel(32);
        let res = replay_cached_streaming_response(
            &cached,
            Some(&reg),
            ToolCallPolicy::Auto,
            &EventSink::new(&tx),
        );
        assert!(res.is_ok(), "replay must succeed");
        drop(tx);
        let events = drain(&mut rx);

        // Concat the deltas by kind. The splitter may emit each kind
        // in one or more chunks (tail buffering across the marker
        // boundary); contract is on the concatenated text + ordering.
        let mut reasoning_concat = String::new();
        let mut content_concat = String::new();
        let mut first_reasoning_idx: Option<usize> = None;
        let mut first_content_idx: Option<usize> = None;
        for (i, ev) in events.iter().enumerate() {
            match ev {
                GenerationEvent::Delta {
                    kind: DeltaKind::Reasoning,
                    text,
                } => {
                    if first_reasoning_idx.is_none() {
                        first_reasoning_idx = Some(i);
                    }
                    reasoning_concat.push_str(text);
                }
                GenerationEvent::Delta {
                    kind: DeltaKind::Content,
                    text,
                } => {
                    if first_content_idx.is_none() {
                        first_content_idx = Some(i);
                    }
                    content_concat.push_str(text);
                }
                _ => {}
            }
        }

        assert_eq!(
            reasoning_concat, "let me compute 2+2",
            "reasoning slot must capture body between <think>...</think> markers \
             (markers themselves swallowed); got events: {events:?}"
        );
        assert_eq!(
            content_concat, "The answer is 4.",
            "content slot must capture post-marker text only; got events: {events:?}"
        );
        // Decision #21 ordering: reasoning streams BEFORE content for
        // Open WebUI's panel UX.
        assert!(
            first_reasoning_idx < first_content_idx,
            "iter C ordering contract violated: reasoning must precede content \
             in event stream; reasoning_idx={first_reasoning_idx:?}, \
             content_idx={first_content_idx:?}, events={events:?}"
        );
        // No raw markers leak.
        for marker in &["<think>", "</think>"] {
            assert!(
                !reasoning_concat.contains(marker),
                "splitter regression: reasoning slot contains raw marker {marker:?}"
            );
            assert!(
                !content_concat.contains(marker),
                "splitter regression: content slot contains raw marker {marker:?}"
            );
        }
        // Last event is Done.
        assert!(
            matches!(events.last(), Some(GenerationEvent::Done { .. })),
            "Done must be terminal event; got events: {events:?}"
        );
    }

    /// Iter C edge case: only reasoning, no post-reasoning content.
    /// Some thinking-mode prompts result in `<think>...</think>` followed
    /// by EOS — the answer is implicit in the reasoning. The replay must
    /// emit reasoning, no content delta, then Done.
    #[test]
    fn replay_streaming_origin_pure_reasoning_no_content() {
        let reg = match super::super::registry::find_for("qwen3.6-27b-dwq46") {
            Some(r) => r,
            None => {
                eprintln!("qwen35 registration absent; skipping iter C pure-reasoning test");
                return;
            }
        };
        if !reg.has_reasoning() {
            return;
        }

        let cached_text = "<think>only thinking</think>";
        let cached = GenerationResult {
            text: cached_text.into(),
            reasoning_text: None,
            prompt_tokens: 3,
            completion_tokens: 4,
            reasoning_tokens: None,
            finish_reason: "stop",
            prefill_duration: Duration::ZERO,
            decode_duration: Duration::ZERO,
            cached_tokens: 3,
            logprobs: None,
        };

        let (tx, mut rx) = mpsc::channel(16);
        let res = replay_cached_streaming_response(
            &cached,
            Some(&reg),
            ToolCallPolicy::Auto,
            &EventSink::new(&tx),
        );
        assert!(res.is_ok());
        drop(tx);
        let events = drain(&mut rx);

        let reasoning_concat: String = events
            .iter()
            .filter_map(|e| match e {
                GenerationEvent::Delta {
                    kind: DeltaKind::Reasoning,
                    text,
                } => Some(text.clone()),
                _ => None,
            })
            .collect();
        let content_concat: String = events
            .iter()
            .filter_map(|e| match e {
                GenerationEvent::Delta {
                    kind: DeltaKind::Content,
                    text,
                } => Some(text.clone()),
                _ => None,
            })
            .collect();

        assert_eq!(reasoning_concat, "only thinking");
        assert_eq!(
            content_concat, "",
            "pure-reasoning input must NOT emit any content delta; got events: {events:?}"
        );
        assert!(matches!(events.last(), Some(GenerationEvent::Done { .. })));
    }

    /// Iter C + iter B-2 interleaving: a reasoning span FOLLOWED BY a
    /// tool-call span must route into `Reasoning` deltas, then `Content`
    /// deltas (preamble post-reasoning), then `ToolCallDelta` events for
    /// the tool-call span — the OpenAI spec mandates `reasoning_content`
    /// and `tool_calls` are mutually exclusive on the same delta chunk.
    /// This locks in the composition contract: ReasoningSplitter runs
    /// FIRST, the Content-classified output then flows through
    /// ToolCallSplitter.
    ///
    /// Uses Qwen 3.5/3.6 markers so both reasoning + tool-call markers
    /// are present in the registration: reasoning `<think>`/`</think>`,
    /// tool-call `<tool_call>`/`</tool_call>`.
    #[test]
    fn replay_routes_reasoning_then_tool_call_in_correct_order() {
        let reg = match super::super::registry::find_for("qwen3.6-27b-dwq46") {
            Some(r) => r,
            None => {
                eprintln!("qwen35 registration absent; skipping iter C+B-2 interleave test");
                return;
            }
        };
        if !reg.has_reasoning() {
            return;
        }
        let (open, close) = match (reg.tool_open, reg.tool_close) {
            (Some(o), Some(c)) => (o, c),
            _ => {
                eprintln!("qwen35 has no tool markers; skipping interleave test");
                return;
            }
        };

        // Streaming-origin shape: full pre-split stream including BOTH
        // reasoning markers AND tool-call markers. Body shape doesn't
        // need to parse as a real tool call — the assertion is on
        // event-class ordering (Reasoning → Content → ToolCallDelta-or-
        // Content-fallback → Done), not on parser outcome.
        let cached_text = format!(
            "<think>I should call the weather tool</think>Let me check. \
             {open}<function=get_weather>\n<parameter=city>\nParis\n</parameter>\n</function>{close}"
        );
        let cached = GenerationResult {
            text: cached_text,
            reasoning_text: None,
            prompt_tokens: 6,
            completion_tokens: 20,
            reasoning_tokens: None,
            finish_reason: "stop",
            prefill_duration: Duration::ZERO,
            decode_duration: Duration::ZERO,
            cached_tokens: 6,
            logprobs: None,
        };

        let (tx, mut rx) = mpsc::channel(64);
        let res = replay_cached_streaming_response(
            &cached,
            Some(&reg),
            ToolCallPolicy::Auto,
            &EventSink::new(&tx),
        );
        assert!(res.is_ok());
        drop(tx);
        let events = drain(&mut rx);

        // Find the index of the FIRST event of each kind.
        let first_reasoning_idx = events.iter().position(|e| {
            matches!(
                e,
                GenerationEvent::Delta {
                    kind: DeltaKind::Reasoning,
                    ..
                }
            )
        });
        let first_content_idx = events.iter().position(|e| {
            matches!(
                e,
                GenerationEvent::Delta {
                    kind: DeltaKind::Content,
                    ..
                }
            )
        });
        let first_tool_call_idx = events
            .iter()
            .position(|e| matches!(e, GenerationEvent::ToolCallDelta { .. }));

        // Reasoning MUST appear; it's unconditional in this fixture.
        let r_idx = first_reasoning_idx.unwrap_or_else(|| {
            panic!(
                "iter C interleave contract: reasoning delta MUST be emitted \
                 for input containing <think>...</think>; got events: {events:?}"
            )
        });
        // Content MUST appear (the "Let me check. " preamble between
        // </think> and the tool-call open marker).
        let c_idx = first_content_idx.unwrap_or_else(|| {
            panic!(
                "iter C interleave contract: content delta MUST be emitted for \
                 the post-reasoning preamble; got events: {events:?}"
            )
        });

        // Decision #21 ordering: reasoning before content.
        assert!(
            r_idx < c_idx,
            "iter C ordering: reasoning ({r_idx}) MUST precede content ({c_idx}); \
             events: {events:?}"
        );

        // If a ToolCallDelta fired (the body parsed under Auto), it MUST
        // come AFTER the reasoning AND after the first content delta —
        // it cannot interleave inside the reasoning span (otherwise the
        // ReasoningSplitter→ToolCallSplitter composition is broken).
        if let Some(t_idx) = first_tool_call_idx {
            assert!(
                r_idx < t_idx,
                "iter C+B-2 composition: reasoning ({r_idx}) MUST precede \
                 tool-call delta ({t_idx}); the ReasoningSplitter runs FIRST \
                 in the engine pipeline. events: {events:?}"
            );
            assert!(
                c_idx < t_idx,
                "iter C+B-2 composition: post-reasoning content ({c_idx}) MUST \
                 precede tool-call delta ({t_idx}); events: {events:?}"
            );
        }

        // OpenAI spec: NO single delta event may carry BOTH
        // reasoning_content AND tool_calls. The Rust enum makes this
        // structurally impossible at the GenerationEvent level — Reasoning
        // deltas are `GenerationEvent::Delta { kind: Reasoning, ... }`,
        // tool deltas are `GenerationEvent::ToolCallDelta { ... }` —
        // distinct variants, both encode through `sse.rs:166-247` into
        // separate JSON chunks. Lock in by asserting NO ToolCallDelta
        // appears at an index ≤ the last Reasoning delta index.
        let last_reasoning_idx = events.iter().rposition(|e| {
            matches!(
                e,
                GenerationEvent::Delta {
                    kind: DeltaKind::Reasoning,
                    ..
                }
            )
        });
        if let (Some(lr), Some(t)) = (last_reasoning_idx, first_tool_call_idx) {
            assert!(
                lr < t,
                "OpenAI spec: tool-call delta MUST NOT precede or interleave \
                 with the reasoning span; last_reasoning={lr}, first_tool_call={t}, \
                 events: {events:?}"
            );
        }

        // Last event is Done.
        assert!(
            matches!(events.last(), Some(GenerationEvent::Done { .. })),
            "Done must terminate stream; events: {events:?}"
        );
    }

    /// Iter C non-streaming-origin replay: when the cache entry was
    /// stored from a non-streaming completion (`reasoning_text=Some(...)`,
    /// `text` post-split), the replay must FIRST emit the explicit
    /// reasoning_text as a Reasoning delta, then route `text` (which
    /// contains NO reasoning markers because they were stripped at
    /// store time) through the splitter as Content. Companion to the
    /// existing `replay_emits_reasoning_then_content_when_reasoning_text_set`
    /// test, but locks in that the **registered model's** ReasoningSplitter
    /// does NOT mistakenly re-classify post-split `text` as containing
    /// reasoning (would cause double-emit).
    #[test]
    fn replay_nonstreaming_origin_does_not_double_emit_reasoning() {
        let reg = match super::super::registry::find_for("qwen3.6-27b-dwq46") {
            Some(r) => r,
            None => {
                eprintln!("qwen35 registration absent; skipping iter C double-emit test");
                return;
            }
        };
        if !reg.has_reasoning() {
            return;
        }

        // Non-streaming-origin shape: post-split text + explicit
        // reasoning_text. Critically, `text` does NOT contain reasoning
        // markers (they were stripped by `split_full_output` at store
        // time). If the splitter mistakenly re-runs and finds nothing,
        // text routes cleanly as Content; if a regression caused it to
        // partially match, we'd see double-emit.
        let cached = cached_non_streaming(
            "The final answer is 42.",
            Some("step 1: parse problem; step 2: compute"),
        );

        let (tx, mut rx) = mpsc::channel(32);
        let res = replay_cached_streaming_response(
            &cached,
            Some(&reg),
            ToolCallPolicy::Auto,
            &EventSink::new(&tx),
        );
        assert!(res.is_ok());
        drop(tx);
        let events = drain(&mut rx);

        // Concat by kind.
        let reasoning_concat: String = events
            .iter()
            .filter_map(|e| match e {
                GenerationEvent::Delta {
                    kind: DeltaKind::Reasoning,
                    text,
                } => Some(text.clone()),
                _ => None,
            })
            .collect();
        let content_concat: String = events
            .iter()
            .filter_map(|e| match e {
                GenerationEvent::Delta {
                    kind: DeltaKind::Content,
                    text,
                } => Some(text.clone()),
                _ => None,
            })
            .collect();

        // Reasoning emitted EXACTLY ONCE — single emission of the
        // stored reasoning_text, not duplicated by a stray splitter
        // match on post-split text.
        assert_eq!(
            reasoning_concat, "step 1: parse problem; step 2: compute",
            "non-streaming-origin: reasoning_text must be emitted verbatim, \
             ONCE. events: {events:?}"
        );
        assert_eq!(
            content_concat, "The final answer is 42.",
            "non-streaming-origin: post-split text must route cleanly as Content. \
             events: {events:?}"
        );
        assert!(matches!(events.last(), Some(GenerationEvent::Done { .. })));
    }

    /// Wave 3.5 HIGH-2 — drain the ReasoningSplitter tail too.
    ///
    /// Cached text contains reasoning markers + a short tail of
    /// content.  Pre-Wave-3.5 the ReasoningSplitter's `finish()` was
    /// never called and the residual tail was lost.
    #[test]
    fn replay_drains_reasoning_splitter_tail() {
        let reg = match super::super::registry::find_for("gemma4-27b-it") {
            Some(r) => r,
            None => {
                eprintln!("gemma4 registration absent; skipping HIGH-2 reasoning drain test");
                return;
            }
        };

        // Build a cached text whose final bytes are a content tail
        // shorter than the reasoning_splitter's tail_cap.  We don't
        // know the exact reasoning markers offline; we just probe the
        // drain semantics by feeding short content with no reasoning.
        // Combined with the tool-call splitter the reasoning drain
        // path is exercised through the registered registration.
        let cached = cached_non_streaming("ok", None);

        let (tx, mut rx) = mpsc::channel(8);
        let res = replay_cached_streaming_response(
            &cached,
            Some(&reg),
            ToolCallPolicy::Auto,
            &EventSink::new(&tx),
        );
        assert!(res.is_ok(), "replay must succeed");
        drop(tx);
        let events = drain(&mut rx);

        let content_concat: String = events
            .iter()
            .filter_map(|e| match e {
                GenerationEvent::Delta {
                    kind: DeltaKind::Content,
                    text,
                } => Some(text.clone()),
                _ => None,
            })
            .collect();
        assert_eq!(
            content_concat, "ok",
            "short content 'ok' MUST traverse both reasoning_splitter \
             AND tool_splitter via the new finish() drain calls; a \
             regression that drops EITHER drain would lose the bytes \
             because both splitters' tail_buf can swallow 2 bytes \
             entirely.\nevents: {events:?}"
        );
    }

    // ────────────────────────────────────────────────────────────────────
    // ADR-005 iter-224 W-A2.3 — fragments-replay branch byte-identity
    // ────────────────────────────────────────────────────────────────────

    /// Falsifiable closure (Worker AA design §6, unit-test variant):
    /// build a known `Vec<CachedFragment>`, store via
    /// `PromptCache::store_with_fragments`, drive the streaming-cache
    /// hit path, capture the replayed event stream, assert
    /// fragment-by-fragment byte-identity.
    ///
    /// **Fail-first**: this test would fail at the start of W-A2.3 (no
    /// fragments branch yet → falls through to splitter-rerun → emits
    /// one big Content delta of `cached.text` instead of the per-token
    /// boundaries the captured Vec preserves).  PASSES post-W-A2.3.
    ///
    /// Covers: Content + Reasoning + ToolCallDelta first-chunk +
    /// ToolCallDelta args-chunk + tool-call finish_reason override
    /// (`saw_tool_call → "tool_calls"`).
    #[test]
    fn streaming_fragment_replay_byte_identical_event_stream() {
        let frags: Vec<CachedFragment> = vec![
            CachedFragment::Reasoning("plan: ".to_string()),
            CachedFragment::Reasoning("call get_weather".to_string()),
            CachedFragment::Content("OK ".to_string()),
            CachedFragment::ToolCallDelta {
                index: 0,
                id: Some("call_hf2q_aabb".to_string()),
                call_type: Some("function".to_string()),
                name: Some("get_weather".to_string()),
                arguments: None,
            },
            CachedFragment::ToolCallDelta {
                index: 0,
                id: None,
                call_type: None,
                name: None,
                arguments: Some("{".to_string()),
            },
            CachedFragment::ToolCallDelta {
                index: 0,
                id: None,
                call_type: None,
                name: None,
                arguments: Some("\"loc\":\"SF\"}".to_string()),
            },
            CachedFragment::Content(" Done.".to_string()),
        ];

        // Cache populated as if streaming origin completed (text is
        // accumulated_text-style; reasoning_text=None because the live
        // splitter routed reasoning into Reasoning deltas as decoded).
        let mut cache = PromptCache::new();
        let tokens: Vec<u32> = vec![100, 200, 300];
        let params = SamplingParams::default();
        let result = GenerationResult {
            text: "<think>plan: call get_weather</think>OK <|tool_call>call:get_weather{loc:<|\"|>SF<|\"|>}<tool_call|> Done.".to_string(),
            reasoning_text: None,
            prompt_tokens: tokens.len(),
            completion_tokens: 10,
            reasoning_tokens: Some(2),
            // Pre-store finish_reason — the replay overrides to
            // "tool_calls" because the captured Vec contains a
            // ToolCallDelta.  This mirrors the live `saw_tool_call`
            // override in `generate_stream_once`.
            finish_reason: "stop",
            prefill_duration: Duration::ZERO,
            decode_duration: Duration::ZERO,
            cached_tokens: 0,
            logprobs: None,
        };
        cache.store_with_fragments(&tokens, &params, &result, Some(frags.clone()));

        // Drive lookup_with_fragments + replay.
        let (cached, cached_frags) = cache
            .lookup_with_fragments(&tokens, &params)
            .expect("greedy hit");
        assert!(
            cached_frags.is_some(),
            "lookup must return Some(fragments) for streaming-origin entry"
        );

        let (tx, mut rx) = mpsc::channel::<GenerationEvent>(64);
        let res = replay_cached_streaming_response_with_fragments(
            &cached,
            None, // registration irrelevant on fragments branch (no splitter run)
            ToolCallPolicy::Auto,
            &EventSink::new(&tx),
            cached_frags,
        );
        assert!(res.is_ok(), "fragments replay must succeed");
        drop(tx);

        let mut emitted: Vec<GenerationEvent> = Vec::new();
        while let Ok(ev) = rx.try_recv() {
            emitted.push(ev);
        }

        // Expected event stream = frags.len() Delta/ToolCallDelta + 1 Done.
        assert_eq!(
            emitted.len(),
            frags.len() + 1,
            "fragments branch emits N fragments + 1 Done; got {} events",
            emitted.len()
        );

        // Per-event byte-identity — Reasoning, Content, ToolCallDelta.
        for (i, frag) in frags.iter().enumerate() {
            match (frag, &emitted[i]) {
                (CachedFragment::Reasoning(t), GenerationEvent::Delta { kind, text }) => {
                    assert_eq!(*kind, DeltaKind::Reasoning);
                    assert_eq!(text, t);
                }
                (CachedFragment::Content(t), GenerationEvent::Delta { kind, text }) => {
                    assert_eq!(*kind, DeltaKind::Content);
                    assert_eq!(text, t);
                }
                (
                    CachedFragment::ToolCallDelta {
                        index: fi,
                        id: fid,
                        call_type: fct,
                        name: fn_,
                        arguments: fargs,
                    },
                    GenerationEvent::ToolCallDelta {
                        index,
                        id,
                        call_type,
                        name,
                        arguments,
                    },
                ) => {
                    assert_eq!(*fi, *index);
                    assert_eq!(fid, id);
                    assert_eq!(fct, call_type);
                    assert_eq!(fn_, name);
                    assert_eq!(fargs, arguments);
                }
                (frag, ev) => panic!("frag[{i}] {frag:?} did not match emitted event {ev:?}"),
            }
        }

        // Terminal Done — finish_reason="tool_calls" (override) +
        // cache-hit signal populated.
        match emitted.last() {
            Some(GenerationEvent::Done {
                finish_reason,
                prompt_tokens,
                completion_tokens,
                stats,
            }) => {
                assert_eq!(
                    *finish_reason, "tool_calls",
                    "fragments-branch saw_tool_call MUST override stored finish_reason"
                );
                assert_eq!(*prompt_tokens, 3);
                assert_eq!(*completion_tokens, 10);
                assert_eq!(stats.cached_prompt_tokens, Some(3));
                assert_eq!(stats.prefill_time_secs, Some(0.0));
                assert_eq!(stats.decode_time_secs, Some(0.0));
                assert_eq!(stats.reasoning_tokens, Some(2));
            }
            other => panic!("last event must be Done; got {other:?}"),
        }
    }

    /// Regression-pin (Chesterton's fence): with `fragments=None`, the
    /// replay path MUST run the splitter pipeline AND drain `tail_buf`
    /// — Wave-3.5 HIGH-2 fix at engine.rs:4332.  Pre-Wave-3.5 the
    /// replay fed `cached.text` once and emitted Done, never calling
    /// `finish()` on either splitter, so held-back tail bytes were
    /// silently dropped.
    ///
    /// W-A2.3 must NOT regress this: a tail-bytes drop on the
    /// non-fragment path would silently truncate cache hits whose
    /// origin was non-streaming (or whose fragments slot is otherwise
    /// `None`).  The existing
    /// `streaming_prompt_cache_replay_tests::replay_drains_*` tests
    /// pin this; this test re-pins specifically the fragments=None
    /// branch with a fresh assertion that the Wave-3.5 drain still
    /// fires.
    #[test]
    fn fragments_none_replay_preserves_splitter_drain() {
        // Build a cached entry with no fragments — should hit
        // splitter-rerun branch.  Use registration so splitters are
        // active (otherwise drain is a no-op).
        let mut cache = PromptCache::new();
        let tokens: Vec<u32> = vec![1, 2, 3];
        let params = SamplingParams::default();

        // Text whose tail is shorter than the splitter's `tail_cap` —
        // pre-Wave-3.5 this would be silently dropped.
        let result = GenerationResult {
            text: "ok".to_string(),
            reasoning_text: None,
            prompt_tokens: tokens.len(),
            completion_tokens: 1,
            reasoning_tokens: None,
            finish_reason: "stop",
            prefill_duration: Duration::ZERO,
            decode_duration: Duration::ZERO,
            cached_tokens: 0,
            logprobs: None,
        };
        cache.store(&tokens, &params, &result);
        // Confirm fragments=None (legacy single-arg store).
        assert!(cache.fragments.is_none());

        let (cached, cached_frags) = cache
            .lookup_with_fragments(&tokens, &params)
            .expect("greedy hit");
        assert!(
            cached_frags.is_none(),
            "non-streaming-origin must yield fragments=None"
        );

        let (tx, mut rx) = mpsc::channel::<GenerationEvent>(32);
        let res = replay_cached_streaming_response_with_fragments(
            &cached,
            Some(&super::super::registry::GEMMA4),
            ToolCallPolicy::Auto,
            &EventSink::new(&tx),
            cached_frags,
        );
        assert!(res.is_ok());
        drop(tx);

        let mut emitted: Vec<GenerationEvent> = Vec::new();
        while let Ok(ev) = rx.try_recv() {
            emitted.push(ev);
        }
        // Expected: splitter pipeline produces a Content delta (2-byte
        // tail flushed via finish() drain) + Done.  If the drain
        // regressed, "ok" would not appear in any Content delta.
        let content_concat: String = emitted
            .iter()
            .filter_map(|ev| match ev {
                GenerationEvent::Delta {
                    kind: DeltaKind::Content,
                    text,
                } => Some(text.clone()),
                _ => None,
            })
            .collect();
        assert_eq!(
            content_concat, "ok",
            "fragments=None branch MUST exercise splitter drain — \
             a regression here re-introduces the Wave-3.5 HIGH-2 \
             tail-drop bug. emitted={emitted:?}"
        );
    }
}

// ---------------------------------------------------------------------------
// Wave-2.5 A1 — conditional grammar wire unit tests
// ---------------------------------------------------------------------------

#[cfg(test)]
mod test_a1_conditional_grammar_wire {
    /// Verify ToolCallSplitter state transitions that drive the A1 grammar gate.
    ///
    /// The grammar mask in the decode loop reads `tool_splitter.in_tool_call()`.
    /// This test confirms the splitter correctly transitions:
    ///   - before any input:     in_tool_call == false  (mask should NOT fire)
    ///   - after ToolCallOpen:   in_tool_call == true   (mask SHOULD fire)
    ///   - after ToolCallClose:  in_tool_call == false  (mask should NOT fire)
    #[test]
    fn splitter_in_body_transitions_drive_grammar_gate() {
        // Use the Gemma4 registration (has real tool open/close markers).
        let reg = crate::serve::api::registry::find_for("gemma4-27b-it")
            .expect("gemma4 registration must exist");
        let (open, close) = match (reg.tool_open, reg.tool_close) {
            (Some(o), Some(c)) => (o, c),
            _ => {
                eprintln!("gemma4 has no tool markers — skip A1 splitter test");
                return;
            }
        };
        let mut splitter = crate::serve::api::registry::ToolCallSplitter::from_registration(&reg)
            .expect("ToolCallSplitter::from_registration must return Some for gemma4");

        // Initial state: not inside a tool-call body.
        // Grammar mask should NOT be active.
        assert!(
            !splitter.in_tool_call(),
            "A1: before any input, in_tool_call must be false \
             (grammar mask must NOT fire for preamble tokens)"
        );

        // Feed the open marker — splitter enters the body.
        // Grammar mask SHOULD now be active.
        let events_open = splitter.feed(open);
        assert!(
            events_open
                .iter()
                .any(|e| matches!(e, crate::serve::api::registry::ToolCallEvent::ToolCallOpen)),
            "A1: feeding the open marker must emit ToolCallOpen"
        );
        assert!(
            splitter.in_tool_call(),
            "A1: after feeding the open marker, in_tool_call must be true \
             (grammar mask MUST fire for body tokens)"
        );

        // Feed the close marker — splitter exits the body.
        // Grammar mask should NOT be active.
        let events_close = splitter.feed(close);
        assert!(
            events_close
                .iter()
                .any(|e| matches!(e, crate::serve::api::registry::ToolCallEvent::ToolCallClose)),
            "A1: feeding the close marker must emit ToolCallClose"
        );
        assert!(
            !splitter.in_tool_call(),
            "A1: after feeding the close marker, in_tool_call must be false \
             (grammar mask must NOT fire after the body)"
        );
    }

    /// Wave 2.6 W-α5 Q2 — replacement for the wave-2.5
    /// `grammar_active_atomic_bool_transitions` test.
    ///
    /// The wave-2.5 architecture used a sibling `Arc<AtomicBool>
    /// grammar_active` toggled by ToolCallOpen/Close.  The audit caught
    /// it as architecturally wrong (mask + advance + dead-check could
    /// disagree because they read different state).  Wave 2.6 moves the
    /// gate INSIDE GrammarRuntime via `awaiting_trigger` — the production
    /// streaming worker now wires `route_content`'s ToolCallOpen handler
    /// to call `runtime.trigger()` directly.  This test exercises that
    /// exact production path: a real registered tool-call splitter, a
    /// real GrammarRuntime, and the same trigger-on-open pattern
    /// `route_content` uses.
    #[test]
    fn tool_call_open_triggers_grammar_runtime() {
        use crate::serve::api::grammar::parser::parse;
        use crate::serve::api::grammar::GrammarRuntime;

        // Real Gemma4 registration with real open/close markers.
        let reg = crate::serve::api::registry::find_for("gemma4-27b-it")
            .expect("gemma4 registration must exist");
        let (open, close) = match (reg.tool_open, reg.tool_close) {
            (Some(o), Some(c)) => (o, c),
            _ => {
                eprintln!("gemma4 has no tool markers — skip Q2 trigger test");
                return;
            }
        };
        let mut splitter = crate::serve::api::registry::ToolCallSplitter::from_registration(&reg)
            .expect("ToolCallSplitter::from_registration must return Some for gemma4");

        // Build a real GrammarRuntime in the lazy state, the way
        // generate_stream_once does for `GrammarKind::ToolCallBodyAuto`.
        let g = parse("root ::= \"x\"\n").expect("parse");
        let rid = g.rule_id("root").expect("root rule");
        let mut runtime = GrammarRuntime::new(g, rid).expect("runtime");
        runtime.set_awaiting_trigger(true);
        assert!(
            runtime.is_awaiting_trigger(),
            "lazy-grammar runtime starts in awaiting_trigger=true (production setup for ToolCallBody-kind requests)"
        );

        // Pre-open: feeding splitter with content that doesn't include
        // the open marker emits no ToolCallOpen and so the production
        // code does NOT call runtime.trigger().  The runtime stays
        // suspended.
        let _events = splitter.feed("plain preamble text ");
        assert!(
            runtime.is_awaiting_trigger(),
            "preamble fragments MUST NOT trigger the runtime"
        );

        // Production trigger pattern (mirrors route_content's
        // ToolCallOpen branch in generate_stream_once):
        let events_open = splitter.feed(open);
        if events_open
            .iter()
            .any(|e| matches!(e, crate::serve::api::registry::ToolCallEvent::ToolCallOpen))
        {
            runtime.trigger();
        }
        assert!(
            !runtime.is_awaiting_trigger(),
            "after ToolCallOpen the production code MUST flip the runtime trigger"
        );

        // Post-open: feeding the close marker through the splitter
        // does NOT reset the runtime — single-call termination comes
        // from the grammar SHAPE exhausting (`body close space` under
        // iter-218's `parallel_tool_calls=false` default), and multi-call
        // re-entry is via the `(call)*` recursion when operators opt
        // into parallel calls (research-report.md Q2; see
        // `/opt/llama.cpp/docs/function-calling.md:24` and
        // `/opt/llama.cpp/common/chat.cpp:1399-1416`).
        let _events_close = splitter.feed(close);
        assert!(
            !runtime.is_awaiting_trigger(),
            "ToolCallClose MUST NOT re-arm the runtime trigger \
             (llama.cpp parity: PR #9639 lazy grammar is one-shot per request; \
              iter-218 narrows the bug class via `parallel_tool_calls=false` default \
              so the bounded shape `body close space` exhausts naturally)"
        );
    }
}

// ---------------------------------------------------------------------------
// Wave 2.8 W-θ HIGH-1 — finalize_streaming_tool_state
//
// Audit-driver tests: exercise the EXACT streaming SSE event chain through
// `tool_splitter.finish()` drain + post-drain Constrained no-call check.
// Tests drive the production helper directly with a real `mpsc::channel`,
// drain the receiver, and assert SSE event shape — NOT a stand-in
// reconstruction (the wave-2 sham-test pattern Codex caught).
// ---------------------------------------------------------------------------
#[cfg(test)]
mod finalize_streaming_tool_state_tests {
    use super::*;
    use crate::serve::api::registry::{self, ToolCallSplitter};
    use crate::serve::api::sse::{DeltaKind, GenerationEvent};
    use tokio::sync::mpsc;

    fn gemma4_reg() -> registry::ModelRegistration {
        registry::find_for("gemma4-27b-it").expect("gemma4 registration must exist")
    }

    /// Drain the receiver synchronously (we are inside a single-threaded
    /// helper that uses `blocking_send`; the matching consumer is
    /// `try_recv` after the helper returns).
    fn drain_recv(rx: &mut mpsc::Receiver<GenerationEvent>) -> Vec<GenerationEvent> {
        let mut out = Vec::new();
        while let Ok(ev) = rx.try_recv() {
            out.push(ev);
        }
        out
    }

    /// Streaming Constrained mid-call truncation: the splitter has seen the
    /// open marker but not the close marker, so `finish()` returns
    /// `ToolCallText(residual)`. Under Constrained policy the helper MUST
    /// emit `GenerationEvent::Error("tool_call_truncated_under_constrained")`
    /// and return `ErrorEmitted` (so the streaming driver skips Done). It
    /// MUST NOT emit Content (the silent-fallback wave-2.6 audit divergence).
    #[test]
    fn streaming_constrained_mid_call_truncation_yields_error_event() {
        let reg = gemma4_reg();
        let mut splitter = ToolCallSplitter::from_registration(&reg)
            .expect("gemma4 has tool markers, splitter must build");

        // Drive the splitter the same way the engine does: feed bytes that
        // include the open marker and a partial body (no close marker).
        // After this, splitter.in_tool_call() == true and tail_buf holds
        // the partial body.
        let open = reg.tool_open.expect("gemma4 has tool_open");
        let _ = splitter.feed(&format!("{open}call:get_weather{{"));
        assert!(
            splitter.in_tool_call(),
            "splitter must be in_tool_call after open marker; finish() will \
             then return ToolCallText (mid-call truncation)"
        );

        let (tx, mut rx) = mpsc::channel::<GenerationEvent>(8);

        let action = finalize_streaming_tool_state(
            Some(&mut splitter),
            ToolCallPolicy::Constrained,
            /* saw_tool_call */ false,
            Some(&reg),
            /* completion_tokens */ 7,
            /* accumulated_text_len */ 18,
            &EventSink::new(&tx),
        );

        assert_eq!(
            action,
            FinalizeStreamingAction::ErrorEmitted,
            "Constrained + ToolCallText residual MUST return ErrorEmitted"
        );

        // Close the sender so try_recv terminates cleanly.
        drop(tx);
        let events = drain_recv(&mut rx);

        assert_eq!(
            events.len(),
            1,
            "exactly one error event expected, got: {:?}",
            events
        );
        match &events[0] {
            GenerationEvent::Error(code) => {
                assert_eq!(
                    code, "tool_call_truncated_under_constrained",
                    "structured error code must match defensive 500 vocabulary"
                );
            }
            other => panic!(
                "expected GenerationEvent::Error, got: {:?} \
                 (silent Content fallback would be the wave-2.6 audit divergence)",
                other
            ),
        }
    }

    /// Streaming Constrained no-call: `saw_tool_call == false` and the
    /// splitter has nothing buffered (decode finished without ever entering
    /// a tool-call span). Under Constrained policy the helper MUST emit
    /// `GenerationEvent::Error("tool_call_no_call_under_constrained")` and
    /// return `ErrorEmitted` BEFORE Done. Mirrors the non-streaming check
    /// in handlers.rs:410-444 (commit da545d5).
    #[test]
    fn streaming_constrained_no_call_yields_error_event() {
        let reg = gemma4_reg();
        let mut splitter = ToolCallSplitter::from_registration(&reg)
            .expect("gemma4 has tool markers, splitter must build");
        // No feed → splitter idle, finish() returns None.
        assert!(
            !splitter.in_tool_call(),
            "splitter must be idle for the no-call test"
        );

        let (tx, mut rx) = mpsc::channel::<GenerationEvent>(8);

        let action = finalize_streaming_tool_state(
            Some(&mut splitter),
            ToolCallPolicy::Constrained,
            /* saw_tool_call */ false,
            Some(&reg),
            /* completion_tokens */ 64,
            /* accumulated_text_len */ 0,
            &EventSink::new(&tx),
        );

        assert_eq!(
            action,
            FinalizeStreamingAction::ErrorEmitted,
            "Constrained + saw_tool_call=false MUST return ErrorEmitted"
        );

        drop(tx);
        let events = drain_recv(&mut rx);

        assert_eq!(events.len(), 1, "exactly one error event expected");
        match &events[0] {
            GenerationEvent::Error(code) => {
                assert_eq!(
                    code, "tool_call_no_call_under_constrained",
                    "structured error code must match defensive 500 vocabulary"
                );
            }
            other => panic!("expected GenerationEvent::Error, got: {:?}", other),
        }
    }

    /// Auto policy mid-call truncation: Auto allows partial / malformed
    /// tool-call syntax. The helper MUST emit the residual as `Content`
    /// (with the literal open marker re-prepended for diagnostic clarity)
    /// and return `Continue` (caller emits Done normally). This is the
    /// pre-2.8 behaviour we MUST preserve.
    #[test]
    fn streaming_auto_mid_call_truncation_emits_content_fallback() {
        let reg = gemma4_reg();
        let mut splitter = ToolCallSplitter::from_registration(&reg).unwrap();
        let open = reg.tool_open.expect("gemma4 has tool_open");
        let _ = splitter.feed(&format!("{open}call:get_weather{{"));
        assert!(splitter.in_tool_call());

        let (tx, mut rx) = mpsc::channel::<GenerationEvent>(8);

        let action = finalize_streaming_tool_state(
            Some(&mut splitter),
            ToolCallPolicy::Auto,
            /* saw_tool_call */ false,
            Some(&reg),
            /* completion_tokens */ 7,
            /* accumulated_text_len */ 18,
            &EventSink::new(&tx),
        );
        assert_eq!(
            action,
            FinalizeStreamingAction::Continue,
            "Auto policy MUST preserve pre-2.8 Content-fallback behaviour"
        );
        drop(tx);
        let events = drain_recv(&mut rx);
        assert_eq!(events.len(), 1, "exactly one Content delta expected");
        match &events[0] {
            GenerationEvent::Delta { kind, text } => {
                assert!(matches!(kind, DeltaKind::Content));
                assert!(
                    text.contains(open),
                    "Auto fallback re-prepends the literal open marker for \
                     diagnostic clarity (so the operator sees the truncation \
                     in delta.content); got: {text:?}"
                );
            }
            other => panic!("expected Content delta, got: {:?}", other),
        }
    }

    /// Auto policy no-call: a turn that never produced a tool call is the
    /// normal Auto outcome. The helper MUST return `Continue` and emit no
    /// events.
    #[test]
    fn streaming_auto_no_call_emits_no_events() {
        let reg = gemma4_reg();
        let mut splitter = ToolCallSplitter::from_registration(&reg).unwrap();
        // Idle splitter, no feed.

        let (tx, mut rx) = mpsc::channel::<GenerationEvent>(8);

        let action = finalize_streaming_tool_state(
            Some(&mut splitter),
            ToolCallPolicy::Auto,
            /* saw_tool_call */ false,
            Some(&reg),
            64,
            0,
            &EventSink::new(&tx),
        );
        assert_eq!(action, FinalizeStreamingAction::Continue);
        drop(tx);
        let events = drain_recv(&mut rx);
        assert!(
            events.is_empty(),
            "Auto + idle splitter must emit no finalize events; got {:?}",
            events
        );
    }

    /// Wave 3 W-B2 — AutoLazyGrammar mid-call truncation MUST yield the
    /// SAME loud-error event as Constrained.
    ///
    /// Under AutoLazyGrammar the per-model body grammar is active inside
    /// the tool-call span (post-trigger). A truncation past ToolCallOpen
    /// without ToolCallClose means decoding stopped mid-grammar — the
    /// runtime is neither accepted nor dead. Same regression signature
    /// as Constrained truncation; same `tool_call_truncated_under_constrained`
    /// error code (preserves the structured-vocabulary single source of
    /// truth so log/metrics matchers continue to work unchanged).
    #[test]
    fn streaming_auto_lazy_grammar_mid_call_truncation_yields_error_event() {
        let reg = gemma4_reg();
        let mut splitter = ToolCallSplitter::from_registration(&reg)
            .expect("gemma4 has tool markers, splitter must build");

        let open = reg.tool_open.expect("gemma4 has tool_open");
        let _ = splitter.feed(&format!("{open}call:get_weather{{"));
        assert!(
            splitter.in_tool_call(),
            "splitter must be in_tool_call after open marker; finish() will \
             then return ToolCallText (mid-call truncation)"
        );

        let (tx, mut rx) = mpsc::channel::<GenerationEvent>(8);

        let action = finalize_streaming_tool_state(
            Some(&mut splitter),
            ToolCallPolicy::AutoLazyGrammar,
            /* saw_tool_call */ false,
            Some(&reg),
            /* completion_tokens */ 7,
            /* accumulated_text_len */ 18,
            &EventSink::new(&tx),
        );

        assert_eq!(
            action,
            FinalizeStreamingAction::ErrorEmitted,
            "AutoLazyGrammar + ToolCallText residual MUST return ErrorEmitted \
             identically to Constrained — the lazy grammar IS active inside \
             the body so a truncation is a regression"
        );

        drop(tx);
        let events = drain_recv(&mut rx);

        assert_eq!(
            events.len(),
            1,
            "exactly one error event expected, got: {:?}",
            events
        );
        match &events[0] {
            GenerationEvent::Error(code) => {
                assert_eq!(
                    code, "tool_call_truncated_under_constrained",
                    "structured error code MUST match the unified vocabulary; \
                     AutoLazyGrammar reuses the Constrained code so log/metrics \
                     matchers continue to work unchanged"
                );
            }
            GenerationEvent::Delta {
                kind: DeltaKind::Content,
                text,
            } => {
                panic!(
                    "REGRESSION: AutoLazyGrammar mid-call truncation emitted \
                     Content fallback (text={text:?}); the wave-3 W-B2 T2.4 \
                     final closure MUST promote this to Error"
                );
            }
            other => panic!(
                "expected GenerationEvent::Error, got: {:?} \
                 (silent Content fallback would be the wave-2.6 audit divergence)",
                other
            ),
        }
    }

    /// Wave 3 W-B2 — AutoLazyGrammar with NO call must NOT trigger the
    /// no-call check.
    ///
    /// Auto explicitly permits the model to emit zero tool calls
    /// (preamble freedom — the whole point of lazy grammar). A streaming
    /// run that ended without ever firing `ToolCallOpen` is the
    /// legitimate Auto-no-call path under AutoLazyGrammar, NOT a
    /// regression.  This is the key semantic distinction from
    /// `ToolCallPolicy::Constrained` (where the eager grammar's
    /// OneOrMoreCalls root mandates >= 1 call).
    #[test]
    fn streaming_auto_lazy_grammar_no_call_emits_no_events() {
        let reg = gemma4_reg();
        let mut splitter = ToolCallSplitter::from_registration(&reg).unwrap();
        // Idle splitter, no feed.
        assert!(!splitter.in_tool_call());

        let (tx, mut rx) = mpsc::channel::<GenerationEvent>(8);

        let action = finalize_streaming_tool_state(
            Some(&mut splitter),
            ToolCallPolicy::AutoLazyGrammar,
            /* saw_tool_call */ false,
            Some(&reg),
            64,
            0,
            &EventSink::new(&tx),
        );
        assert_eq!(
            action,
            FinalizeStreamingAction::Continue,
            "AutoLazyGrammar + idle splitter MUST return Continue — Auto \
             explicitly allows the model to emit zero tool calls (preamble \
             freedom is the whole point of lazy grammar). The no-call check \
             stays Constrained-only."
        );
        drop(tx);
        let events = drain_recv(&mut rx);
        assert!(
            events.is_empty(),
            "AutoLazyGrammar + idle splitter must emit no finalize events; \
             got {:?}",
            events
        );
    }

    /// Constrained policy with `saw_tool_call == true` (the model produced
    /// at least one full call): no-call check MUST NOT fire even though
    /// policy is Constrained. The helper returns `Continue`.
    #[test]
    fn streaming_constrained_saw_tool_call_continues_to_done() {
        let reg = gemma4_reg();
        let mut splitter = ToolCallSplitter::from_registration(&reg).unwrap();
        // Drive a full call so splitter is idle and would have emitted a
        // ToolCallClose during streaming. We only care that
        // splitter.finish() returns None (no residual) and that the
        // post-drain no-call check sees saw_tool_call=true.
        let open = reg.tool_open.expect("open");
        let close = reg.tool_close.expect("close");
        let _ = splitter.feed(&format!("{open}call:foo{{}}{close}"));
        assert!(!splitter.in_tool_call());

        let (tx, mut rx) = mpsc::channel::<GenerationEvent>(8);

        let action = finalize_streaming_tool_state(
            Some(&mut splitter),
            ToolCallPolicy::Constrained,
            /* saw_tool_call */ true,
            Some(&reg),
            12,
            18,
            &EventSink::new(&tx),
        );
        assert_eq!(action, FinalizeStreamingAction::Continue);
        drop(tx);
        let events = drain_recv(&mut rx);
        assert!(
            events.is_empty(),
            "Constrained + saw_tool_call=true must emit no finalize events; \
             got {:?}",
            events
        );
    }
}

// ---------------------------------------------------------------------------
// Wave 3 W-A3 — emit_streaming_tool_call_close (T2.4 partial removal)
// Wave 3 W-B2 — emit_streaming_tool_call_close (T2.4 final closure on
//               registered Auto-with-tools path)
//
// Audit-driver tests for the body-parse-failure branches of
// `emit_streaming_tool_call_close`.  Three scenarios:
//
//   1. `required_body_parse_failure_yields_error_not_content` (W-A3) —
//      Constrained policy + parse failure → GenerationEvent::Error
//      ("tool_call_unreachable_fallback_required"). MUST NOT emit Content.
//
//   2. `auto_lazy_grammar_body_parse_failure_yields_error_not_content`
//      (W-B2) — AutoLazyGrammar policy + parse failure →
//      GenerationEvent::Error("tool_call_unreachable_fallback_required").
//      MUST NOT emit Content.  Same loud-error promotion as Constrained
//      because the lazy grammar IS active inside the body.
//
//   3. `auto_body_parse_failure_preserves_content_fallback` (W-A3) —
//      Auto (no grammar) policy + parse failure →
//      GenerationEvent::Delta{Content, body_dump}.  Regression-preserve:
//      the content fallback for unconstrained Auto (no tools[] / unknown
//      family) is the defined behaviour and MUST NOT regress.
//
// All tests drive `emit_streaming_tool_call_close` directly with
// `parsed = None` (simulating a malformed body after ToolCallClose fires).
// ---------------------------------------------------------------------------
#[cfg(test)]
mod emit_streaming_tool_call_close_tests {
    use super::*;
    use crate::serve::api::sse::{DeltaKind, GenerationEvent};
    use tokio::sync::mpsc;

    fn drain_recv(rx: &mut mpsc::Receiver<GenerationEvent>) -> Vec<GenerationEvent> {
        let mut out = Vec::new();
        while let Ok(ev) = rx.try_recv() {
            out.push(ev);
        }
        out
    }

    /// T2.4 partial removal — Required path.
    ///
    /// Simulates: ToolCallText has accumulated malformed JSON ("garbage{{}}")
    /// into `body`, then ToolCallClose fires. `parse_tool_call_body` returns
    /// None.  Under Constrained policy `emit_streaming_tool_call_close` MUST:
    ///   - return `Err(())`
    ///   - emit exactly one `GenerationEvent::Error` with code
    ///     `"tool_call_unreachable_fallback_required"`
    ///   - NOT emit any `GenerationEvent::Delta { kind: Content, … }`
    ///
    /// This branch should be unreachable in correct operation (the eager
    /// grammar from wave-2.7 W-η da545d5 physically prevents a bad body).
    /// If it fires, it is a grammar-engine regression and must be loud.
    #[test]
    fn required_body_parse_failure_yields_error_not_content() {
        let (tx, mut rx) = mpsc::channel::<GenerationEvent>(8);
        let mut tc_index: usize = 0;
        let mut saw_tc: bool = false;

        let result = emit_streaming_tool_call_close(
            None, // parsed = None: simulates a body that failed parse_tool_call_body
            "garbage{{}}".to_string(),
            ToolCallPolicy::Constrained,
            &mut tc_index,
            &mut saw_tc,
            &EventSink::new(&tx),
        );

        assert!(
            result.is_err(),
            "Constrained + parse failure MUST return Err(()) \
             (streaming driver aborts decode loop)"
        );
        assert_eq!(
            tc_index, 0,
            "tc_index must not be incremented on parse failure"
        );
        assert!(!saw_tc, "saw_tc must remain false on parse failure");

        drop(tx);
        let events = drain_recv(&mut rx);

        assert_eq!(
            events.len(),
            1,
            "exactly one GenerationEvent::Error expected; got: {:?}",
            events
        );
        match &events[0] {
            GenerationEvent::Error(code) => {
                assert_eq!(
                    code, "tool_call_unreachable_fallback_required",
                    "error code MUST be 'tool_call_unreachable_fallback_required' \
                     (wave 3 W-A3 T2.4 partial removal); old 'tool_call_parse_failure' \
                     code would indicate a regression to wave-2.5 A4 vocabulary"
                );
            }
            GenerationEvent::Delta {
                kind: DeltaKind::Content,
                text,
            } => {
                panic!(
                    "REGRESSION: Constrained parse failure emitted Content fallback \
                     (text={text:?}); this is the T2.4 silent-fallback that W-A3 removes"
                );
            }
            other => panic!("expected GenerationEvent::Error, got: {:?}", other),
        }
    }

    /// Wave 3 W-B2 — T2.4 FINAL closure for the registered-Auto path.
    ///
    /// Simulates the same malformed body under
    /// `ToolCallPolicy::AutoLazyGrammar` — the policy the handler sets
    /// when `tool_choice=auto` AND the W-B2 lazy grammar IS active
    /// (tools[] non-empty AND model family registered AND
    /// `effective_grammar_kind == ToolCallBodyAuto`).
    /// `emit_streaming_tool_call_close` MUST treat this branch
    /// identically to Constrained:
    ///   - return `Err(())`
    ///   - emit exactly one `GenerationEvent::Error` with code
    ///     `"tool_call_unreachable_fallback_required"`
    ///   - NOT emit any `GenerationEvent::Delta { kind: Content, … }`
    ///
    /// Rationale: under AutoLazyGrammar the per-model body grammar is
    /// active inside the tool-call span (the `awaiting_trigger` flag is
    /// flipped by `route_content`'s ToolCallOpen handler before the
    /// body bytes are accepted by the runtime). A parse failure
    /// therefore means the lazy grammar engine produced structurally
    /// invalid output — same regression signature as Constrained.
    #[test]
    fn auto_lazy_grammar_body_parse_failure_yields_error_not_content() {
        let (tx, mut rx) = mpsc::channel::<GenerationEvent>(8);
        let mut tc_index: usize = 0;
        let mut saw_tc: bool = false;

        let result = emit_streaming_tool_call_close(
            None,
            "garbage{{}}".to_string(),
            ToolCallPolicy::AutoLazyGrammar,
            &mut tc_index,
            &mut saw_tc,
            &EventSink::new(&tx),
        );

        assert!(
            result.is_err(),
            "AutoLazyGrammar + parse failure MUST return Err(()) \
             (streaming driver aborts decode loop, identical to Constrained)"
        );
        assert_eq!(
            tc_index, 0,
            "tc_index must not be incremented on parse failure"
        );
        assert!(!saw_tc, "saw_tc must remain false on parse failure");

        drop(tx);
        let events = drain_recv(&mut rx);

        assert_eq!(
            events.len(),
            1,
            "exactly one GenerationEvent::Error expected; got: {:?}",
            events
        );
        match &events[0] {
            GenerationEvent::Error(code) => {
                assert_eq!(
                    code, "tool_call_unreachable_fallback_required",
                    "AutoLazyGrammar must emit the SAME error code as Constrained \
                     (the unified loud-error vocabulary)"
                );
            }
            GenerationEvent::Delta {
                kind: DeltaKind::Content,
                text,
            } => {
                panic!(
                    "REGRESSION: AutoLazyGrammar parse failure emitted Content fallback \
                     (text={text:?}); the wave-3 W-B2 T2.4 final closure MUST promote \
                     this branch to Error identically to Constrained"
                );
            }
            other => panic!("expected GenerationEvent::Error, got: {:?}", other),
        }
    }

    /// T2.4 regression-preserve — Auto (no grammar) path.
    ///
    /// Simulates the same malformed body under
    /// `ToolCallPolicy::Auto` — the policy the handler sets when
    /// `tool_choice=auto` AND no grammar is active (no tools[] declared,
    /// OR an unregistered model family). This branch MUST:
    ///   - return `Ok(())`
    ///   - emit exactly one `GenerationEvent::Delta { kind: Content, text: body_dump }`
    ///   - NOT emit `GenerationEvent::Error`
    ///
    /// Under Auto-no-grammar there is no enforcement on body shape. The
    /// model may legitimately emit partial / malformed tool-call syntax;
    /// preserving the content fallback lets the client see the raw bytes
    /// rather than losing them. Wave 3 W-B2 narrowed this branch (the
    /// registered-family+tools path now uses `AutoLazyGrammar`), but the
    /// remaining Auto-no-grammar slice still keeps the fallback — this
    /// test pins it.
    #[test]
    fn auto_body_parse_failure_preserves_content_fallback() {
        let body = "some malformed body text".to_string();
        let (tx, mut rx) = mpsc::channel::<GenerationEvent>(8);
        let mut tc_index: usize = 0;
        let mut saw_tc: bool = false;

        let result = emit_streaming_tool_call_close(
            None, // parsed = None: simulates a body that failed parse_tool_call_body
            body.clone(),
            ToolCallPolicy::Auto,
            &mut tc_index,
            &mut saw_tc,
            &EventSink::new(&tx),
        );

        assert!(
            result.is_ok(),
            "Auto + parse failure MUST return Ok(()) \
             (content fallback is the defined Auto behaviour, not an error)"
        );
        assert_eq!(
            tc_index, 0,
            "tc_index must not be incremented on parse failure"
        );
        assert!(!saw_tc, "saw_tc must remain false on parse failure");

        drop(tx);
        let events = drain_recv(&mut rx);

        assert_eq!(
            events.len(),
            1,
            "exactly one Content delta expected for Auto fallback; got: {:?}",
            events
        );
        match &events[0] {
            GenerationEvent::Delta { kind, text } => {
                assert!(
                    matches!(kind, DeltaKind::Content),
                    "Auto parse-failure delta MUST be DeltaKind::Content; got: {:?}",
                    kind
                );
                assert_eq!(
                    text, &body,
                    "Auto fallback MUST re-emit the original body_dump verbatim; \
                     got: {text:?}"
                );
            }
            GenerationEvent::Error(code) => {
                panic!(
                    "REGRESSION: Auto parse failure promoted to GenerationEvent::Error \
                     (code={code:?}); Auto MUST preserve content fallback until \
                     Wave 3 Phase B lazy grammar lands"
                );
            }
            other => panic!("expected Content delta, got: {:?}", other),
        }
    }
}

// ---------------------------------------------------------------------------
// Wave 3 W-B3 — T2.3 incremental tool-call argument streaming.
//
// Audit-driver tests for `ToolCallStreamEmitter`: they feed body fragments
// through `advance` and `finalize` directly, then assert the emitted SSE
// shape exactly matches the OpenAI Chat Completions streaming spec:
//
//   - Chunk 1: function.name complete, no arguments.
//   - Chunks 2..N-1: arguments fragments that concatenate to valid JSON.
//   - Final chunk: closing `}` (and any kv tail the streaming scanner
//     deferred). On the streaming-driver side `finish_reason="tool_calls"`
//     fires from the terminating `Done` event after `saw_tool_call` is
//     latched true by `finalize`.
//
// All tests drive the emitter directly with hand-crafted body fragments
// (rather than through the full `ToolCallSplitter` + `route_content`
// pipeline) so the tail-parser + chunk-emit logic is isolated from
// splitter / grammar / sampler concerns.
// ---------------------------------------------------------------------------
#[cfg(test)]
mod tool_call_stream_emitter_tests {
    use super::*;
    use crate::serve::api::sse::GenerationEvent;
    use tokio::sync::mpsc;

    fn drain(rx: &mut mpsc::Receiver<GenerationEvent>) -> Vec<GenerationEvent> {
        let mut out = Vec::new();
        while let Ok(ev) = rx.try_recv() {
            out.push(ev);
        }
        out
    }

    /// Collapse a sequence of `ToolCallDelta` events into the (name, args_string)
    /// pair the OpenAI client would reconstruct: name from the first chunk that
    /// carries it, args from the concatenation of every chunk's `arguments`.
    fn rebuild_call(events: &[GenerationEvent], expect_index: usize) -> (Option<String>, String) {
        let mut name: Option<String> = None;
        let mut args = String::new();
        for ev in events {
            if let GenerationEvent::ToolCallDelta {
                index,
                name: n,
                arguments,
                ..
            } = ev
            {
                if *index != expect_index {
                    continue;
                }
                if let Some(nm) = n {
                    name = Some(nm.clone());
                }
                if let Some(a) = arguments {
                    args.push_str(a);
                }
            }
        }
        (name, args)
    }

    /// `streaming_tool_call_emits_name_in_first_chunk` — chunk 1 carries
    /// `function.name` complete, with `arguments=None`. Subsequent chunks
    /// stream `arguments` only (no `name` retransmission).
    #[test]
    fn streaming_tool_call_emits_name_in_first_chunk() {
        let (tx, mut rx) = mpsc::channel::<GenerationEvent>(32);
        let mut emitter = ToolCallStreamEmitter::new(Some("gemma4"), 0);
        // Feed a body prefix that contains the name + opening brace, plus
        // a started kv. The first `advance` call MUST emit chunk 1
        // (id+type+name) and chunk 2 (the args opening `{`).
        let body = "call:get_weather{location:<|\"|>San Fra".to_string();
        emitter
            .advance(&body, &EventSink::new(&tx))
            .expect("advance ok");
        drop(tx);
        let events = drain(&mut rx);
        // Chunk 1: name+id+type, no args.
        match &events[0] {
            GenerationEvent::ToolCallDelta {
                index,
                id,
                call_type,
                name,
                arguments,
            } => {
                assert_eq!(*index, 0);
                assert!(id.is_some(), "first chunk MUST carry id");
                assert_eq!(call_type.as_deref(), Some("function"));
                assert_eq!(name.as_deref(), Some("get_weather"));
                assert!(
                    arguments.is_none(),
                    "first chunk MUST NOT carry arguments (name-only per spec)"
                );
            }
            other => panic!("expected first ToolCallDelta with name; got {other:?}"),
        }
        // Chunk 2: args opening `{`, no name.
        match &events[1] {
            GenerationEvent::ToolCallDelta {
                name,
                arguments,
                id,
                ..
            } => {
                assert!(id.is_none(), "subsequent chunks MUST NOT retransmit id");
                assert!(name.is_none(), "subsequent chunks MUST NOT retransmit name");
                assert_eq!(
                    arguments.as_deref(),
                    Some("{"),
                    "second chunk MUST be the args opening `{{`"
                );
            }
            other => panic!("expected ToolCallDelta with `{{` arg; got {other:?}"),
        }
    }

    /// `streaming_tool_call_emits_arguments_incrementally` — feed a 3-fragment
    /// body and assert at least 3 distinct `arguments` deltas fire (one per
    /// closed-kv boundary).
    #[test]
    fn streaming_tool_call_emits_arguments_incrementally() {
        let (tx, mut rx) = mpsc::channel::<GenerationEvent>(32);
        let mut emitter = ToolCallStreamEmitter::new(Some("gemma4"), 0);
        let mut body = String::new();
        // Fragment 1: header + first kv started, no closer.
        body.push_str("call:get_weather{location:<|\"|>");
        emitter.advance(&body, &EventSink::new(&tx)).expect("frag1");
        // Fragment 2: close first kv with `,` and start second kv.
        body.push_str("San Francisco<|\"|>,");
        emitter.advance(&body, &EventSink::new(&tx)).expect("frag2");
        // Fragment 3: second kv complete + closer.
        body.push_str("units:<|\"|>celsius<|\"|>}");
        emitter.advance(&body, &EventSink::new(&tx)).expect("frag3");
        // Close finalizes the last kv + `}`.
        let mut tc_index: usize = 0;
        let mut saw_tc: bool = false;
        emitter
            .finalize(
                body,
                Some(&super::super::registry::GEMMA4),
                ToolCallPolicy::Constrained,
                &mut tc_index,
                &mut saw_tc,
                &EventSink::new(&tx),
            )
            .expect("finalize");
        drop(tx);
        let events = drain(&mut rx);
        // Count `arguments`-bearing deltas. We expect at least:
        //   chunk: `{`   (opening, from advance frag1)
        //   chunk: `"location":"San Francisco"`  (frag2 closes first kv)
        //   chunk: `,"units":"celsius"`  +  closing `}` (finalize)
        //     OR finalize emits both as a single tail.
        let arg_chunks: Vec<&str> = events
            .iter()
            .filter_map(|ev| {
                if let GenerationEvent::ToolCallDelta {
                    arguments: Some(a), ..
                } = ev
                {
                    Some(a.as_str())
                } else {
                    None
                }
            })
            .collect();
        assert!(
            arg_chunks.len() >= 3,
            "expected >=3 arguments deltas (incremental shape); got {arg_chunks:?}"
        );
        assert_eq!(tc_index, 1, "tc_index MUST be incremented on finalize");
        assert!(saw_tc, "saw_tc MUST be latched true on finalize");
    }

    /// `streaming_tool_call_arguments_concatenate_to_valid_json` — collect
    /// every `arguments` delta in stream order, concatenate them, JSON-parse
    /// the result, and assert it equals the canonical `parse_tool_call_body`
    /// args output.
    #[test]
    fn streaming_tool_call_arguments_concatenate_to_valid_json() {
        let (tx, mut rx) = mpsc::channel::<GenerationEvent>(64);
        let mut emitter = ToolCallStreamEmitter::new(Some("gemma4"), 0);
        // Feed body in 4 fragments that bisect the kv structure at
        // non-boundary points.
        let mut body = String::new();
        let chunks = [
            "call:get_weather{location:<|\"|>",
            "San Francis",
            "co<|\"|>,units:<|\"|>celsius<|\"|>",
            "}",
        ];
        for c in &chunks {
            body.push_str(c);
            emitter
                .advance(&body, &EventSink::new(&tx))
                .expect("advance");
        }
        let mut tc_index: usize = 0;
        let mut saw_tc: bool = false;
        emitter
            .finalize(
                body.clone(),
                Some(&super::super::registry::GEMMA4),
                ToolCallPolicy::Constrained,
                &mut tc_index,
                &mut saw_tc,
                &EventSink::new(&tx),
            )
            .expect("finalize");
        drop(tx);
        let events = drain(&mut rx);
        let (name, args) = rebuild_call(&events, 0);
        assert_eq!(name.as_deref(), Some("get_weather"));
        let parsed: serde_json::Value =
            serde_json::from_str(&args).expect("args MUST be valid JSON");
        // Canonical args from the existing parser:
        let canonical =
            super::super::registry::parse_tool_call_body(&super::super::registry::GEMMA4, &body)
                .expect("canonical parse");
        let canonical_json: serde_json::Value =
            serde_json::from_str(&canonical.arguments_json).expect("canonical json");
        assert_eq!(
            parsed, canonical_json,
            "concatenated streaming args MUST equal canonical parse"
        );
    }

    /// `streaming_tool_call_emits_finish_reason_tool_calls_terminal` — verify
    /// `finalize` latches `saw_tc=true` so the Done event downstream picks
    /// `finish_reason="tool_calls"`. The terminating `Done` is a downstream
    /// concern (driven by the decode loop and `replay_cached_streaming_response`
    /// branch); here we pin the contract that finalize-on-success MUST set
    /// `saw_tc` so the Done-emit logic at engine.rs:2513 + replay.rs:2538 can
    /// override the default `"stop"` to `"tool_calls"`.
    #[test]
    fn streaming_tool_call_emits_finish_reason_tool_calls_terminal() {
        let (tx, mut rx) = mpsc::channel::<GenerationEvent>(16);
        let mut emitter = ToolCallStreamEmitter::new(Some("gemma4"), 0);
        let body = "call:f{x:1}".to_string();
        emitter
            .advance(&body, &EventSink::new(&tx))
            .expect("advance");
        let mut tc_index: usize = 0;
        let mut saw_tc: bool = false;
        emitter
            .finalize(
                body,
                Some(&super::super::registry::GEMMA4),
                ToolCallPolicy::Constrained,
                &mut tc_index,
                &mut saw_tc,
                &EventSink::new(&tx),
            )
            .expect("finalize");
        drop(tx);
        assert!(
            saw_tc,
            "finalize on success MUST latch saw_tc=true so the Done event \
             picks finish_reason=\"tool_calls\""
        );
        assert_eq!(tc_index, 1, "tc_index MUST advance to 1");
        let events = drain(&mut rx);
        let last = events
            .iter()
            .rev()
            .find_map(|ev| {
                if let GenerationEvent::ToolCallDelta {
                    arguments: Some(a), ..
                } = ev
                {
                    Some(a.as_str())
                } else {
                    None
                }
            })
            .expect("at least one arguments delta");
        assert!(
            last.ends_with('}'),
            "the final arguments delta MUST close the JSON object with `}}`; \
             got tail={last:?}"
        );
    }

    /// `streaming_multiple_tool_calls_with_distinct_indices` — drive two
    /// emitters in sequence (mirrors `parallel_tool_calls=true` where
    /// the model emits two consecutive `<|tool_call>...<tool_call|>` spans),
    /// and assert each call's deltas carry distinct `index` values.
    #[test]
    fn streaming_multiple_tool_calls_with_distinct_indices() {
        let (tx, mut rx) = mpsc::channel::<GenerationEvent>(64);
        let mut tc_index: usize = 0;
        let mut saw_tc: bool = false;

        // Call 0.
        let mut em0 = ToolCallStreamEmitter::new(Some("gemma4"), tc_index);
        let body0 = "call:f0{a:1}".to_string();
        em0.advance(&body0, &EventSink::new(&tx)).expect("advance0");
        em0.finalize(
            body0,
            Some(&super::super::registry::GEMMA4),
            ToolCallPolicy::Constrained,
            &mut tc_index,
            &mut saw_tc,
            &EventSink::new(&tx),
        )
        .expect("finalize0");
        assert_eq!(tc_index, 1, "tc_index advances after call 0");

        // Call 1.
        let mut em1 = ToolCallStreamEmitter::new(Some("gemma4"), tc_index);
        let body1 = "call:f1{b:2}".to_string();
        em1.advance(&body1, &EventSink::new(&tx)).expect("advance1");
        em1.finalize(
            body1,
            Some(&super::super::registry::GEMMA4),
            ToolCallPolicy::Constrained,
            &mut tc_index,
            &mut saw_tc,
            &EventSink::new(&tx),
        )
        .expect("finalize1");
        assert_eq!(tc_index, 2, "tc_index advances after call 1");

        drop(tx);
        let events = drain(&mut rx);
        let mut indices = std::collections::BTreeSet::new();
        for ev in &events {
            if let GenerationEvent::ToolCallDelta { index, .. } = ev {
                indices.insert(*index);
            }
        }
        assert!(
            indices.contains(&0) && indices.contains(&1),
            "both index=0 and index=1 MUST appear in the delta stream; got {indices:?}"
        );
    }

    /// Wave 3.5 MED — `streaming_single_fragment_emits_incremental_shape`.
    ///
    /// Honest replacement for the misnamed
    /// `streaming_single_fragment_falls_back_to_close_buffered_shape`
    /// test (Wave 3 W-B3).  The previous name promised a "legacy
    /// fallback" to the pre-W-B3 two-chunk close-buffered shape, but
    /// the actual `advance` + `finalize` flow emits MORE than two
    /// chunks even for a single-fragment body:
    ///
    ///   * `advance(body)` sees `call:f{` complete in the FIRST call
    ///     (engine.rs:2196-2239) and immediately emits chunk 1
    ///     (id+name, no arguments) and chunk 2 (`{` opening).
    ///   * `finalize` then emits the residual tail
    ///     (`"x":1` + closing `}`) as additional chunks.
    ///
    /// `finalize` only delegates to the legacy
    /// `emit_streaming_tool_call_close` when `name_emitted == false`
    /// (engine.rs:2398-2406) — i.e. when `advance` couldn't extract
    /// the name from any prefix (unknown family OR the single
    /// fragment didn't contain enough to find the name).  For a
    /// well-formed Gemma 4 single-fragment body like `call:f{x:1}`,
    /// `advance` extracts `f` immediately, sets `name_emitted=true`,
    /// and the legacy fallback is NEVER taken.
    ///
    /// Wave 3 audit divergence "W-B3 single-fragment fallback"
    /// severity MED at
    /// `/tmp/cfa-cfa-20260427-adr005-wave3/codex-review-last.txt`:
    ///
    ///   "advance emits name and the arguments opening as soon as it
    ///    sees call:f{ at engine.rs:2196-2239; finalize delegates to
    ///    legacy only if name_emitted is false at engine.rs:2398-2406.
    ///    The test named streaming_single_fragment_falls_back_to_
    ///    close_buffered_shape only checks concatenated JSON, not
    ///    event count or legacy shape."
    ///
    /// Resolution per audit recommendation (ii) + worker prompt
    /// directive: the incremental shape IS the canonical OpenAI
    /// spec; the "single-fragment legacy fallback" was an unnecessary
    /// backwards-compat hack that was never actually wired up for
    /// well-formed bodies.  Update test to assert the true shape:
    /// chunk 1 has id+name, chunk 2 has `{` opening, finalize emits
    /// the tail (multiple kv chunks possible if the kv-scanner ran;
    /// or one tail chunk if it didn't).  Concatenated arguments MUST
    /// be valid JSON.  No legacy two-chunk shape is preserved or
    /// expected.
    ///
    /// The TRUE legacy fallback (delegating to
    /// `emit_streaming_tool_call_close`) is exercised by
    /// `streaming_unknown_family_falls_back_to_legacy` (unknown
    /// family → `advance` is a no-op → `finalize` delegates).
    #[test]
    fn streaming_single_fragment_emits_incremental_shape() {
        let (tx, mut rx) = mpsc::channel::<GenerationEvent>(16);
        let mut emitter = ToolCallStreamEmitter::new(Some("gemma4"), 0);
        // A single fragment containing the FULL body.  The emitter's
        // first `advance` extracts the name `f` and emits:
        //   chunk 1: id + type + name (no arguments)
        //   chunk 2: arguments=`{`
        // Then finalize emits the residual tail.
        let body = "call:f{x:1}".to_string();
        emitter
            .advance(&body, &EventSink::new(&tx))
            .expect("advance");
        let mut tc_index: usize = 0;
        let mut saw_tc: bool = false;
        emitter
            .finalize(
                body,
                Some(&super::super::registry::GEMMA4),
                ToolCallPolicy::Constrained,
                &mut tc_index,
                &mut saw_tc,
                &EventSink::new(&tx),
            )
            .expect("finalize");
        drop(tx);
        let events = drain(&mut rx);

        // Honest event-shape assertion (audit-driven).  All emitted
        // events MUST be ToolCallDelta with the canonical incremental
        // shape — NOT the legacy two-chunk close-buffered shape.
        assert!(
            events
                .iter()
                .all(|e| matches!(e, GenerationEvent::ToolCallDelta { .. })),
            "all events MUST be ToolCallDelta (no Content fallback for \
             well-formed Gemma 4 body); got {events:?}"
        );

        // Chunk 1 MUST carry id+type+name (no arguments).  This is the
        // canonical OpenAI streaming first-chunk shape.
        let chunk1 = events.first().expect("at least one event");
        match chunk1 {
            GenerationEvent::ToolCallDelta {
                index,
                id,
                call_type,
                name,
                arguments,
            } => {
                assert_eq!(*index, 0, "chunk 1 index MUST be 0");
                assert!(
                    id.is_some(),
                    "chunk 1 MUST carry id (canonical OpenAI shape)"
                );
                assert_eq!(call_type.as_deref(), Some("function"));
                assert_eq!(
                    name.as_deref(),
                    Some("f"),
                    "chunk 1 MUST carry function name"
                );
                assert!(arguments.is_none(), "chunk 1 MUST NOT carry arguments");
            }
            other => panic!("chunk 1 must be ToolCallDelta with id+name; got {other:?}"),
        }

        // Chunk 2 MUST be the `{` opening (advance step 2).  No id, no
        // name retransmission.
        let chunk2 = events.get(1).expect("at least two events");
        match chunk2 {
            GenerationEvent::ToolCallDelta {
                index,
                id,
                call_type,
                name,
                arguments,
            } => {
                assert_eq!(*index, 0);
                assert!(id.is_none(), "chunk 2 MUST NOT retransmit id");
                assert!(call_type.is_none(), "chunk 2 MUST NOT retransmit type");
                assert!(name.is_none(), "chunk 2 MUST NOT retransmit name");
                assert_eq!(
                    arguments.as_deref(),
                    Some("{"),
                    "chunk 2 MUST be the args opening `{{`"
                );
            }
            other => panic!("chunk 2 must be ToolCallDelta with `{{`; got {other:?}"),
        }

        // Event count MUST be at least 2 (chunks 1 and 2 from advance).
        // The pre-W-B3 legacy two-chunk close-buffered shape would have
        // been: chunk 1 (id+name+full args), chunk 2 (close).  The
        // Wave 3 W-B3 incremental shape is strictly different and
        // typically emits more chunks (one per closed kv + a tail).
        assert!(
            events.len() >= 2,
            "incremental shape emits at least 2 chunks (id+name then `{{`); \
             got {} events: {events:?}",
            events.len()
        );

        // Concatenated args across all chunks MUST be valid JSON
        // matching the input body.  This is the canonical OpenAI
        // accumulator-on-the-client contract.
        let (name, args) = rebuild_call(&events, 0);
        assert_eq!(name.as_deref(), Some("f"));
        let v: serde_json::Value =
            serde_json::from_str(&args).expect("args concatenate to valid JSON");
        assert_eq!(v, serde_json::json!({"x": 1}));

        // tc_index MUST advance and saw_tc latch — these are the
        // contracts the live decode loop relies on.
        assert_eq!(tc_index, 1, "tc_index MUST advance to 1 after finalize");
        assert!(saw_tc, "saw_tc MUST latch true after finalize");
    }

    /// Qwen 3.5/3.6 streaming — `<function=NAME>...<parameter=KEY>VAL</parameter>...</function>`
    /// emits one delta per closed `<parameter>` block.
    #[test]
    fn streaming_qwen35_emits_per_parameter_block() {
        let (tx, mut rx) = mpsc::channel::<GenerationEvent>(32);
        let mut emitter = ToolCallStreamEmitter::new(Some("qwen35"), 0);
        let mut body = String::new();
        body.push_str("<function=lookup>");
        emitter.advance(&body, &EventSink::new(&tx)).expect("frag1");
        body.push_str("\n<parameter=q>\n\"hello\"\n</parameter>");
        emitter.advance(&body, &EventSink::new(&tx)).expect("frag2");
        body.push_str("\n<parameter=k>\n5\n</parameter>\n</function>");
        emitter.advance(&body, &EventSink::new(&tx)).expect("frag3");
        let mut tc_index: usize = 0;
        let mut saw_tc: bool = false;
        emitter
            .finalize(
                body.clone(),
                Some(&super::super::registry::QWEN35),
                ToolCallPolicy::Constrained,
                &mut tc_index,
                &mut saw_tc,
                &EventSink::new(&tx),
            )
            .expect("finalize");
        drop(tx);
        let events = drain(&mut rx);
        let (name, args) = rebuild_call(&events, 0);
        assert_eq!(name.as_deref(), Some("lookup"));
        let v: serde_json::Value =
            serde_json::from_str(&args).expect("args concatenate to valid JSON");
        let canonical =
            super::super::registry::parse_tool_call_body(&super::super::registry::QWEN35, &body)
                .expect("canonical");
        let cv: serde_json::Value = serde_json::from_str(&canonical.arguments_json).unwrap();
        assert_eq!(
            v, cv,
            "Qwen 3.5/3.6 streaming args MUST equal canonical parse"
        );
    }

    /// Unknown family — `advance` is a no-op (no `name_emitted`), and
    /// `finalize` delegates to the legacy close-buffered path. Verify the
    /// emitter never emits anything before finalize when the family lacks a
    /// streaming converter, AND that the legacy `Auto` content fallback
    /// fires when the body fails to parse.
    #[test]
    fn streaming_unknown_family_falls_back_to_legacy() {
        let (tx, mut rx) = mpsc::channel::<GenerationEvent>(8);
        let mut emitter = ToolCallStreamEmitter::new(None, 0);
        emitter
            .advance("anything goes here", &EventSink::new(&tx))
            .expect("advance no-op");
        // No emissions yet — unknown family declined the streaming path.
        let mid_events: Vec<_> = std::iter::from_fn(|| rx.try_recv().ok()).collect();
        assert!(
            mid_events.is_empty(),
            "unknown family MUST NOT emit deltas during advance; got {mid_events:?}"
        );
        // Finalize under Auto policy with no registration — body is treated
        // as malformed (no parser), legacy emit fires the content fallback.
        let mut tc_index: usize = 0;
        let mut saw_tc: bool = false;
        let result = emitter.finalize(
            "anything goes here".to_string(),
            None,
            ToolCallPolicy::Auto,
            &mut tc_index,
            &mut saw_tc,
            &EventSink::new(&tx),
        );
        assert!(
            result.is_ok(),
            "Auto + unparseable MUST be Ok (content fallback)"
        );
        drop(tx);
        let events = drain(&mut rx);
        assert_eq!(events.len(), 1, "exactly one Content delta expected");
        match &events[0] {
            GenerationEvent::Delta {
                kind: super::super::sse::DeltaKind::Content,
                text,
            } => {
                assert_eq!(text, "anything goes here");
            }
            other => panic!("expected Content delta, got {other:?}"),
        }
    }

    // ─── iter-219 reproducers (ADR-005 Phase 4 reopen iter-218 honest-scope) ───
    //
    // iter-218 LIVE testing surfaced a malformed `function.name` of the form
    // `get_currentcall:get_current_weather` on the first ToolCallDelta when a
    // Gemma 4 model emits a leading non-tool-call content fragment ending in
    // `get_current` followed by `<|tool_call>call:get_current_weather{...}<tool_call|>`.
    // The bug is independent of the iter-218 loop fix; it surfaced past the
    // structural unblock. These tests drive the FULL splitter→emitter pipeline
    // exactly the way `route_content` does (engine.rs:4598+) so the hypothesis
    // is testable without a live model. Per the engineering mantra
    // ("Code + test == truth"), the doc-anchored byte stream is the only
    // ground truth we can rely on for a regression test.
    //
    // The flow mirrors route_content:
    //   - Content   → recorded (delta.content)
    //   - ToolCallOpen  → body.clear(), emitter = Some(new)
    //   - ToolCallText  → body.push_str(t), emitter.advance(body, ...)
    //   - ToolCallClose → emitter.finalize(...)

    /// Drive splitter + emitter pipeline through a sequence of decoded
    /// fragments (one entry per token) and return (delta_content_concat,
    /// tool_call_events).
    ///
    /// `policy` selects the close-time fallback shape:
    ///   - `Constrained` / `AutoLazyGrammar`: parse failure raises a loud
    ///     `GenerationEvent::Error` (grammar engine bug surface).
    ///   - `Auto`: parse failure emits `Content(raw_body)` so the malformed
    ///     bytes still reach the client. Use this for iter-219b which
    ///     covers special-token-pollution recovery.
    fn drive_splitter_emitter_pipeline(fragments: &[&str]) -> (String, Vec<GenerationEvent>) {
        drive_splitter_emitter_pipeline_with_policy(fragments, ToolCallPolicy::Constrained)
    }

    fn drive_splitter_emitter_pipeline_with_policy(
        fragments: &[&str],
        policy: ToolCallPolicy,
    ) -> (String, Vec<GenerationEvent>) {
        let (tx, mut rx) = mpsc::channel::<GenerationEvent>(256);
        let reg = &super::super::registry::GEMMA4;
        let mut splitter = super::super::registry::ToolCallSplitter::from_registration(reg)
            .expect("gemma4 has tool markers");
        let mut body = String::new();
        let mut emitter: Option<ToolCallStreamEmitter> = None;
        let mut tc_index: usize = 0;
        let mut saw_tc: bool = false;

        let drive_events = |events: Vec<super::super::registry::ToolCallEvent>,
                            body: &mut String,
                            emitter: &mut Option<ToolCallStreamEmitter>,
                            tc_index: &mut usize,
                            saw_tc: &mut bool,
                            sink: &EventSink<'_>| {
            for ev in events {
                match ev {
                    super::super::registry::ToolCallEvent::Content(t) => {
                        if !t.is_empty() {
                            sink.blocking_send(GenerationEvent::Delta {
                                kind: super::super::sse::DeltaKind::Content,
                                text: t,
                            })
                            .expect("send content");
                        }
                    }
                    super::super::registry::ToolCallEvent::ToolCallOpen => {
                        body.clear();
                        *emitter = Some(ToolCallStreamEmitter::new(Some(reg.family), *tc_index));
                    }
                    super::super::registry::ToolCallEvent::ToolCallText(t) => {
                        body.push_str(&t);
                        if let Some(em) = emitter.as_mut() {
                            em.advance(body, sink).expect("advance");
                        }
                    }
                    super::super::registry::ToolCallEvent::ToolCallClose => {
                        let body_dump = std::mem::take(body);
                        let mut em = emitter.take().unwrap_or_else(|| {
                            ToolCallStreamEmitter::new(Some(reg.family), *tc_index)
                        });
                        // Auto policy: parse failure → content fallback (Ok).
                        // Constrained / AutoLazyGrammar: parse failure → loud
                        // Err(()). Allow either path here so test scenarios
                        // can exercise both contracts.
                        let _ = em.finalize(body_dump, Some(reg), policy, tc_index, saw_tc, sink);
                    }
                }
            }
        };

        let sink = EventSink::new(&tx);
        for frag in fragments {
            let events = splitter.feed(frag);
            drive_events(
                events,
                &mut body,
                &mut emitter,
                &mut tc_index,
                &mut saw_tc,
                &sink,
            );
        }
        if let Some(tail) = splitter.finish() {
            drive_events(
                vec![tail],
                &mut body,
                &mut emitter,
                &mut tc_index,
                &mut saw_tc,
                &sink,
            );
        }
        drop(sink);
        drop(tx);

        let mut content = String::new();
        let mut tool_events = Vec::new();
        for ev in drain(&mut rx) {
            match &ev {
                GenerationEvent::Delta {
                    kind: super::super::sse::DeltaKind::Content,
                    text,
                } => content.push_str(text),
                GenerationEvent::ToolCallDelta { .. } => tool_events.push(ev),
                _ => {}
            }
        }
        (content, tool_events)
    }

    /// iter-219 baseline — single-fragment whole-emission case. The model
    /// emits the full template-shaped sequence in one step; splitter sees
    /// one big string. Establishes that the splitter+emitter is correct
    /// when boundary issues are absent.
    #[test]
    fn iter219_baseline_single_fragment_yields_clean_name() {
        let raw = "<|tool_response>get_current\
                   <|tool_call>call:get_current_weather\
                   {location:<|\"|>Paris<|\"|>}<tool_call|>";
        let (content, tool_events) = drive_splitter_emitter_pipeline(&[raw]);
        let (name, args) = rebuild_call(&tool_events, 0);
        assert_eq!(
            name.as_deref(),
            Some("get_current_weather"),
            "BASELINE: single-fragment whole-emit MUST extract clean name. \
             Got name={name:?}, content={content:?}, args={args:?}"
        );
        let parsed: serde_json::Value =
            serde_json::from_str(&args).expect("args MUST be valid JSON");
        assert_eq!(parsed["location"], "Paris");
    }

    /// iter-219 reproducer — token-boundary case. The Gemma 4 tokenizer
    /// emits the bug-relevant string as the following decoded fragments
    /// (verified against the real `tokenizer.json` round-trip on
    /// 2026-04-30): `<|tool_response>`, `get`, `_`, `current`,
    /// `<|tool_call>`, `call`, `:`, `get`, `_`, `current`, `_`,
    /// `weather`, `{`, `location`, `:`, `<|"|>`, `Paris`, `<|"|>`, `}`,
    /// `<tool_call|>`. This MUST yield the same clean name as the
    /// single-fragment case — anything else is a token-boundary regression
    /// in the splitter / emitter.
    #[test]
    fn iter219_reproducer_token_boundary_yields_clean_name() {
        let fragments: &[&str] = &[
            "<|tool_response>",
            "get",
            "_",
            "current",
            "<|tool_call>",
            "call",
            ":",
            "get",
            "_",
            "current",
            "_",
            "weather",
            "{",
            "location",
            ":",
            "<|\"|>",
            "Paris",
            "<|\"|>",
            "}",
            "<tool_call|>",
        ];
        let (content, tool_events) = drive_splitter_emitter_pipeline(fragments);
        let (name, args) = rebuild_call(&tool_events, 0);
        assert_eq!(
            name.as_deref(),
            Some("get_current_weather"),
            "iter-219: token-boundary feed MUST extract clean name == \
             \"get_current_weather\" (not the malformed \
             \"get_currentcall:get_current_weather\" observed in iter-218 \
             LIVE). Got name={name:?}, content={content:?}, args={args:?}"
        );
        let parsed: serde_json::Value =
            serde_json::from_str(&args).expect("args MUST be valid JSON");
        assert_eq!(parsed["location"], "Paris");
        // The leading `<|tool_response>get_current` must end up in
        // delta.content (or be absorbed elsewhere coherently); critically,
        // it MUST NOT pollute the tool-call body.
        assert!(
            !name.as_deref().unwrap_or("").contains("call:"),
            "iter-219: tool-call name MUST NOT contain `call:` (would \
             indicate body absorbed pre-open content). name={name:?}"
        );
    }

    /// iter-219 stress — leading content WITHOUT the `<|tool_response>`
    /// stray prefix, just a plain `get_current` content fragment before the
    /// open marker (the structural shape of the bug per the ADR-218
    /// honest-scope note).
    #[test]
    fn iter219_reproducer_leading_get_current_content_isolated() {
        let fragments: &[&str] = &[
            "get",
            "_",
            "current",
            "<|tool_call>",
            "call",
            ":",
            "get",
            "_",
            "current",
            "_",
            "weather",
            "{",
            "location",
            ":",
            "<|\"|>",
            "Paris",
            "<|\"|>",
            "}",
            "<tool_call|>",
        ];
        let (content, tool_events) = drive_splitter_emitter_pipeline(fragments);
        let (name, args) = rebuild_call(&tool_events, 0);
        assert_eq!(
            name.as_deref(),
            Some("get_current_weather"),
            "iter-219: leading `get_current` content MUST be routed to \
             delta.content (NOT prepended to the tool body). Got \
             name={name:?}, content={content:?}, args={args:?}"
        );
        // `get_current` should appear in delta.content (the splitter routed
        // it correctly) — present check is loose because exact whitespace
        // is irrelevant; the regression signature is name pollution.
        let parsed: serde_json::Value =
            serde_json::from_str(&args).expect("args MUST be valid JSON");
        assert_eq!(parsed["location"], "Paris");
    }

    /// iter-219b reproducer (LIVE-driven 2026-05-01) — Agent A captured
    /// `name="get_currentcall:get_current_weather"` from a live curl SSE
    /// against scenario_2. The model emitted `<|tool_response>` (token id 50)
    /// MID-tool-call, between two `call:` prefixes. The `ToolCallSplitter`
    /// only recognizes `<|tool_call>` open / `<tool_call|>` close; it has no
    /// awareness of `<|tool_response>` as a span-terminator, so the inner
    /// special-token literal flows through as `ToolCallText` and is appended
    /// to the body buffer verbatim. `extract_gemma4_name_prefix` then runs on
    /// `body == "call:get_current<|tool_response>call:get_current_weather{...}"`
    /// and reads everything up to the first `{` as the name.
    ///
    /// This test reproduces the exact failure mode at the unit level (no
    /// live model needed). It MUST fail on HEAD with the malformed name and
    /// pass after the splitter is taught to treat `<|tool_response>` (and
    /// any other registered Gemma 4 in-call special-token marker) as a
    /// resync that aborts the current call body.
    #[test]
    fn iter219b_reproducer_tool_response_inside_call() {
        let fragments: &[&str] = &[
            "<|tool_call>",
            "call",
            ":",
            "get",
            "_",
            "current",
            "<|tool_response>", // stray special token MID-CALL
            "call",
            ":",
            "get",
            "_",
            "current",
            "_",
            "weather",
            "{",
            "location",
            ":",
            "<|\"|>",
            "Paris",
            "<|\"|>",
            "}",
            "<tool_call|>",
        ];
        // Use Auto policy — the iter-219b fix routes malformed bodies
        // through the content-fallback path (None from
        // `extract_gemma4_name_prefix` → `emit_streaming_tool_call_close`
        // emits `Content(raw_body)`). Constrained policy would also work
        // but raises a loud `GenerationEvent::Error` instead of falling
        // back; we exercise the Auto contract here as the user-facing path.
        let (_content, tool_events) =
            drive_splitter_emitter_pipeline_with_policy(fragments, ToolCallPolicy::Auto);
        let (name, _args) = rebuild_call(&tool_events, 0);
        // Print the actual name so we can see what the splitter+emitter
        // produces under this scenario.
        eprintln!("iter-219b actual name: {name:?}");
        // The structural invariant: the function name MUST NOT be polluted
        // by content emitted before the second `call:` marker. Either the
        // splitter aborts the malformed call (preferred — emit as Content
        // fallback per OpenAI Auto-mode) OR the emitter rejects the
        // malformed-prefix body. Both are valid fixes; both surface as
        // `name != Some("get_currentcall:get_current_weather")`.
        assert_ne!(
            name.as_deref(),
            Some("get_currentcall:get_current_weather"),
            "iter-219b: stray <|tool_response> mid-call MUST NOT pollute the \
             tool-call name. The current implementation absorbs the special \
             token into the body buffer; fix candidates: (a) extend \
             ToolCallSplitter to treat <|tool_response> / <tool_response|> \
             as resync points; (b) sanity-check extract_gemma4_name_prefix \
             rejects names containing special-token characters."
        );
        // Tighter contract: name should NOT contain ANY non-identifier
        // characters (`:`, `<`, `|`, `>` are all special-token bytes). A
        // healthy splitter+emitter MUST yield either a valid identifier or
        // None (call rejected).
        if let Some(n) = name.as_deref() {
            assert!(
                !n.contains(':') && !n.contains('<') && !n.contains('|') && !n.contains('>'),
                "iter-219b: tool-call name must not contain special-token bytes. \
                 Got name={n:?} — body absorbed mid-call special-token literal."
            );
        }
    }

    /// iter-219b second-order test (2026-05-01) — when the validity gate
    /// rejects a malformed name and Auto-policy falls back to emitting the
    /// raw body as Content, the body MUST NOT contain special-token byte
    /// sequences. Otherwise the iter-217-class leak (`<|channel>` /
    /// `<|tool_response>` etc. reaching `delta.content`) re-surfaces via
    /// the fallback path. The fix is to scrub registered Gemma 4 / Qwen
    /// 3.5/3.6 in-call special-token markers from the body before
    /// emitting the content fallback.
    ///
    /// PRE-FIX HEAD: this test FAILS at the `<|tool_response>`
    /// substring assertion because `emit_streaming_tool_call_close` blindly
    /// emits `body_dump` verbatim under the Auto branch.
    /// POST-FIX: scrubbed body emitted; assertions PASS.
    #[test]
    fn iter219b_content_fallback_does_not_leak_special_tokens() {
        let fragments: &[&str] = &[
            "<|tool_call>",
            "call",
            ":",
            "get",
            "_",
            "current",
            "<|tool_response>",
            "call",
            ":",
            "get",
            "_",
            "current",
            "_",
            "weather",
            "{",
            "location",
            ":",
            "<|\"|>",
            "Paris",
            "<|\"|>",
            "}",
            "<tool_call|>",
        ];
        let (content, _tool_events) =
            drive_splitter_emitter_pipeline_with_policy(fragments, ToolCallPolicy::Auto);
        eprintln!("iter-219b content fallback: {content:?}");
        // The body fallback must scrub any registered in-call special-token
        // markers. Listed against the Gemma 4 BUILTIN_REGISTRATIONS family
        // (mirrors `tests/openwebui_multiturn.rs::assert_no_leaked_special_tokens`):
        for marker in &[
            "<|channel>",
            "<channel|>",
            "<|tool_call>",
            "<tool_call|>",
            "<|tool_response>",
            "<tool_response|>",
            "<|turn>",
            "<turn|>",
        ] {
            assert!(
                !content.contains(marker),
                "iter-219b: Auto-policy content-fallback path leaked \
                 special-token marker {marker:?} into delta.content. \
                 Content was: {content:?}"
            );
        }
    }
}

// ---------------------------------------------------------------------------
// ADR-040 Phase C iter-1.5 tests (2026-05-23) — Liskov fix for iter-1
//
// Tests for the EngineMode enum + spawn_with_mode (Result-returning) +
// mode() accessor (echoes stored mode). These tests:
//   1. Pin the public API surface (variant Debug names, Default impl,
//      Copy + Clone + PartialEq + Eq bounds).
//   2. Pin the iter-1.5 Liskov-honest contract — `mode()` returns the
//      mode stored on `EngineInner`, not a hardcoded default.
//   3. Pin the iter-1.5 fail-fast contract — `spawn_with_mode` rejects
//      `SlotAware` with `EngineSpawnError::ModeNotYetWired` rather than
//      silently degrading to SerialFifo (the iter-1 Liskov violation
//      that both adversarial reviewers flagged as CRITICAL).
//   4. Pin the 3-arg `Engine::spawn` signature at the compile-time level —
//      it is the ADR-005 byte-equivalence entry point and may NOT be
//      modified by future iters (iter-2 adds new constructors instead).
//
// Per ADR-040 §3.6 + AC-3 + §7 ("no fallback, no stub"): every byte of
// `Engine` behaviour under `SerialFifo` is bit-equivalent to pre-ADR-040
// and unwired modes fail fast at the API boundary instead of degrading.
// These tests guard the boundary at the type-system level — they do not
// exercise the worker thread (existing `tests` module at line ~7986
// covers the runtime FIFO behaviour and will be the regression target
// when iter-2 forks the SlotAware path).
// ---------------------------------------------------------------------------
#[cfg(test)]
mod adr040_phase_c_iter1_engine_mode_tests {
    use super::*;

    /// AC-3 pin — `EngineMode::default()` is `SerialFifo`. This is the
    /// production-default contract under §3.6: with `HF2Q_SCHEDULER` unset,
    /// the engine behaves byte-for-byte as pre-ADR-040.
    #[test]
    fn engine_mode_default_is_serial_fifo() {
        let mode = EngineMode::default();
        assert!(
            matches!(mode, EngineMode::SerialFifo),
            "ADR-040 §3.6: EngineMode::default() MUST be SerialFifo to \
             preserve the ADR-005 Phase 2 contract. Got {mode:?}."
        );
    }

    /// Pin: `SlotAware { max_slots }` round-trips its payload through Debug.
    /// Ensures the variant carries its capacity bound and that future
    /// refactors don't accidentally strip the inner field.
    #[test]
    fn engine_mode_slot_aware_carries_max_slots() {
        let mode = EngineMode::SlotAware { max_slots: 4 };
        let dbg = format!("{mode:?}");
        assert!(
            dbg.contains("SlotAware"),
            "Debug format must name the variant. Got: {dbg}"
        );
        assert!(
            dbg.contains("max_slots") && dbg.contains('4'),
            "Debug format must round-trip the max_slots payload. Got: {dbg}"
        );
        // Destructure-bind to pin the variant shape — fails to compile if
        // the field name or position changes.
        let EngineMode::SlotAware { max_slots } = mode else {
            panic!("expected SlotAware variant");
        };
        assert_eq!(max_slots, 4);
    }

    /// Compile-time gate — `EngineMode` MUST implement `Copy + Clone +
    /// PartialEq + Eq`. Copy/Clone make it trivially passable to
    /// `spawn_with_mode` by value; PartialEq/Eq let tests + callers use
    /// `assert_eq!` against the mode without falling back to `matches!`.
    /// (PartialEq/Eq added at iter-1.5 per Claude reviewer's
    /// `minor_findings[2]` recommendation.)
    #[test]
    fn engine_mode_is_copy_clone_and_eq() {
        fn assert_copy_clone_eq<T: Copy + Clone + PartialEq + Eq>() {}
        assert_copy_clone_eq::<EngineMode>();

        // Runtime witness: actually exercise both impls.
        let a = EngineMode::SlotAware { max_slots: 8 };
        let b = a; // Copy — `a` still usable.
        let c = a.clone();
        assert!(matches!(a, EngineMode::SlotAware { max_slots: 8 }));
        assert!(matches!(b, EngineMode::SlotAware { max_slots: 8 }));
        assert!(matches!(c, EngineMode::SlotAware { max_slots: 8 }));

        // Eq witness — same variant + same payload compares equal;
        // different payload compares unequal; cross-variant compares
        // unequal.
        assert_eq!(a, EngineMode::SlotAware { max_slots: 8 });
        assert_ne!(a, EngineMode::SlotAware { max_slots: 9 });
        assert_ne!(a, EngineMode::SerialFifo);
        assert_eq!(EngineMode::SerialFifo, EngineMode::default());
    }

    /// Pin: Debug names both variants verbatim. Diagnostics + log lines
    /// will name the mode; the variant names are public surface.
    #[test]
    fn engine_mode_debug_names_variants() {
        let serial = format!("{:?}", EngineMode::SerialFifo);
        assert!(
            serial.contains("SerialFifo"),
            "Debug must name SerialFifo. Got: {serial}"
        );

        let slot = format!("{:?}", EngineMode::SlotAware { max_slots: 1 });
        assert!(
            slot.contains("SlotAware"),
            "Debug must name SlotAware. Got: {slot}"
        );
    }

    /// Compile-time gate — `Engine::spawn_with_mode` exists with the
    /// iter-1.5 `Result`-returning signature. If a future iter renames
    /// the constructor, drops the `EngineMode` parameter, reorders args,
    /// or reverts to the iter-1 infallible signature, this fails to
    /// compile. The Result type is what makes the iter-1 Liskov
    /// violation impossible to silently reintroduce — callers MUST handle
    /// the `Err` arm today.
    ///
    /// The function is NOT called (would require a real `LoadedModel` +
    /// GGUF on disk); the binding alone is the load-bearing assertion.
    #[test]
    fn spawn_with_mode_signature_returns_result() {
        let _f: fn(
            LoadedModel,
            usize,
            Option<u64>,
            EngineMode,
        ) -> std::result::Result<Engine, EngineSpawnError> = Engine::spawn_with_mode;
        // SlotAware variant constructible at this iter (signature-only).
        let _m: EngineMode = EngineMode::SlotAware { max_slots: 4 };
    }

    /// ADR-040 iter-1.5 — `Engine::mode()` returns the mode stored on
    /// `EngineInner`, not a hardcoded default. This is the Liskov-honest
    /// version of the iter-1 accessor; iter-1's "always return
    /// `EngineMode::default()`" was a Liskov-substitution violation
    /// (Codex `critical_findings[0]` + Claude `critical_findings[1]`).
    ///
    /// Compile-only proof: `mode()` returns `EngineMode`. Full
    /// instantiation needs a real LoadedModel + GGUF on disk; verified
    /// at compile time via type signature. An integration test at Phase
    /// C iter-2 will exercise the SlotAware live-route end-to-end and
    /// assert `engine.mode() == EngineMode::SlotAware { max_slots: N }`
    /// after a successful spawn.
    #[test]
    fn mode_accessor_echoes_requested_mode() {
        let _m: fn(&Engine) -> EngineMode = Engine::mode;
    }

    /// ADR-040 iter-1.5 fail-fast contract — `EngineSpawnError::
    /// ModeNotYetWired`'s `Display` impl names both the variant
    /// ("SlotAware") and the iter that lands the runtime ("iter-2").
    /// This test exercises the error path without needing a live
    /// `LoadedModel`; it is the pattern-matched analog of the
    /// behavioural assertion Codex's `critical_findings[0]` requested.
    #[test]
    fn engine_spawn_error_mode_not_yet_wired_names_iters() {
        let err = EngineSpawnError::ModeNotYetWired {
            iter_landed: "C1.5",
            iter_required: "C2",
        };
        let msg = format!("{}", err);
        assert!(msg.contains("SlotAware"), "msg: {}", msg);
        assert!(msg.contains("iter-2"), "msg: {}", msg);
    }

    /// Signature-only pin: proves the existing 3-arg `Engine::spawn`
    /// constructor signature has not changed since pre-ADR-040. This is
    /// NOT a behaviour pin — the spawn body could be silently rewritten
    /// without this test failing. Behavioural byte-equivalence is owned
    /// by Phase C iter-2's live regression test (per ADR-040 §3.6
    /// amended); F4 (renamed from
    /// `engine_spawn_signature_unchanged_at_phase_c_iter_1` per Codex
    /// `major_findings[3]` to make the signature-only nature of the
    /// guard explicit in the test name).
    #[test]
    fn engine_spawn_3_arg_signature_compile_pin() {
        let _spawn: fn(LoadedModel, usize, Option<u64>) -> Engine = Engine::spawn;
    }
}

// ---------------------------------------------------------------------------
// ADR-040 Phase A4 iter-1 (2026-05-30) — spec-decode max-slots threshold gate.
//
// Per the §6.1.53 + §6.1.54 dossier closure (`docs/research/
// adr040-a4-drafter-multi-seq-dossier-2026-05-30.md`), 3 independent
// published sources confirm spec-decode net-regresses above 4-8
// concurrent requests. iter-A4 iter-1 ships:
//   - The MultiSeqDrafterKvCache + alloc + MultiSeqKvCache impl
//     (`src/inference/spec_decode/eagle3/kv_cache.rs`).
//   - The SpecDecodeMaxSlotsAboveBatchedThreshold typed
//     EngineSpawnError variant.
//   - A pre-flight gate in `Engine::spawn_with_mode` that rejects
//     `EngineMode::SlotAware { max_slots: N }` when N > threshold AND
//     `HF2Q_SPEC_DECODE_ALLOW_OVERSIZED != 1`.
//   - Pure env-reader helpers `read_spec_decode_max_batched_slots` +
//     `read_spec_decode_allow_oversized` so tests can deterministically
//     drive policy without touching process env.
//
// H229 pins the threshold-gate behaviour at the structural level:
//   - Pure env-reader parser correctness (default, parse, malformed,
//     overflow, zero-trap).
//   - Typed error shape (variant exists; carries max_slots + threshold
//     + cite static-str; Display includes the dossier path so operator
//     log greps land on the research source).
//   - The constants + helpers are pub so cross-module callers + tests
//     stay deterministic.
//
// Skip-mode pin only — does NOT exercise the worker thread (would need
// a real `LoadedModel` + GGUF on disk).  H229_spawn_arm_rejects_when_
// oversized + H229_spawn_arm_allows_when_opted_in are gated end-to-end
// witnesses tracked in iter-A4-cont-inflection-bench per dossier §6.
// ---------------------------------------------------------------------------
#[cfg(test)]
mod adr040_phase_a4_iter1_spec_decode_threshold_gate_tests {
    use super::*;

    /// **H229 (spec-decode env-reader default)** — when
    /// `HF2Q_SPEC_DECODE_MAX_BATCHED_SLOTS` is unset, the spec-decode
    /// drafter gate defaults to
    /// `ADR040_A4_DEFAULT_SPEC_DECODE_MAX_BATCHED_SLOTS` (= 4, the
    /// conservative dossier §1.5 + §3 lower edge).  This default stays 4
    /// FAIL-CLOSED — the continuous-batching ceiling (8) is the SEPARATE
    /// `ADR040_F_DEFAULT_CONTINUOUS_BATCHING_MAX_SLOTS`
    /// (`adr040_phase_f_gate_decoupling_pin`).
    #[test]
    fn h229_env_reader_default_is_4_when_unset() {
        let threshold = read_spec_decode_max_batched_slots(|_| None);
        assert_eq!(
            threshold, ADR040_A4_DEFAULT_SPEC_DECODE_MAX_BATCHED_SLOTS,
            "H229: env unset MUST return the dossier-cited spec-decode default 4"
        );
        assert_eq!(
            threshold, 4,
            "H229: spec-decode default MUST stay 4 (fail-closed) — the future \
             drafter regresses above 4; continuous batching uses the separate \
             ADR040_F_DEFAULT_CONTINUOUS_BATCHING_MAX_SLOTS=8"
        );
    }

    /// **Phase F gate decoupling pin** (codex `b671dfe0` review item (c)) —
    /// the spec-decode drafter gate and the continuous-batching capacity
    /// gate are SEPARATE constants with DIFFERENT defaults, so a future
    /// drafter implementer cannot inherit the relaxed continuous default
    /// for the actual spec-decode path.  If a refactor ever re-merges them
    /// (makes both equal), this fails LOUDLY.
    #[test]
    fn adr040_phase_f_gate_decoupling_pin() {
        assert_eq!(
            ADR040_A4_DEFAULT_SPEC_DECODE_MAX_BATCHED_SLOTS, 4,
            "spec-decode drafter gate MUST stay 4 (fail-closed; dossier regression > 4)"
        );
        assert_eq!(
            ADR040_F_DEFAULT_CONTINUOUS_BATCHING_MAX_SLOTS, 8,
            "continuous-batching ceiling MUST be 8 (operator request, N=8 proven)"
        );
        assert_ne!(
            ADR040_A4_DEFAULT_SPEC_DECODE_MAX_BATCHED_SLOTS,
            ADR040_F_DEFAULT_CONTINUOUS_BATCHING_MAX_SLOTS,
            "the two gates MUST remain decoupled — re-merging them re-opens the \
             codex-flagged fail-open footgun (drafter inheriting the relaxed 8)"
        );
    }

    /// **Phase F continuous-batching reader** — default 8, `HF2Q_MAX_BATCHED_SLOTS`
    /// preferred, legacy `HF2Q_SPEC_DECODE_MAX_BATCHED_SLOTS` honoured as a
    /// deprecated back-compat fallback.
    #[test]
    fn adr040_phase_f_continuous_batching_reader_default_and_precedence() {
        // Unset → default 8.
        assert_eq!(read_continuous_batching_max_slots(|_| None), 8);
        // Primary env wins.
        assert_eq!(
            read_continuous_batching_max_slots(
                |n| (n == "HF2Q_MAX_BATCHED_SLOTS").then(|| "6".to_string())
            ),
            6
        );
        // Legacy env honoured as fallback when primary unset (back-compat).
        assert_eq!(
            read_continuous_batching_max_slots(
                |n| (n == "HF2Q_SPEC_DECODE_MAX_BATCHED_SLOTS").then(|| "5".to_string())
            ),
            5
        );
        // Primary takes precedence over legacy when BOTH set.
        assert_eq!(
            read_continuous_batching_max_slots(|n| match n {
                "HF2Q_MAX_BATCHED_SLOTS" => Some("8".to_string()),
                "HF2Q_SPEC_DECODE_MAX_BATCHED_SLOTS" => Some("2".to_string()),
                _ => None,
            }),
            8
        );
        // Zero/malformed trap to default.
        assert_eq!(
            read_continuous_batching_max_slots(
                |n| (n == "HF2Q_MAX_BATCHED_SLOTS").then(|| "0".to_string())
            ),
            8
        );
    }

    /// **H229 (env-reader parse)** — well-formed integer values are
    /// parsed verbatim.  Operators who have measured a different
    /// workload-specific inflection point can tune via this env.
    #[test]
    fn h229_env_reader_parses_well_formed_integers() {
        for (env_value, expected) in [("1", 1u32), ("2", 2), ("8", 8), ("100", 100)] {
            let got = read_spec_decode_max_batched_slots(|name| {
                assert_eq!(name, "HF2Q_SPEC_DECODE_MAX_BATCHED_SLOTS");
                Some(env_value.to_string())
            });
            assert_eq!(
                got, expected,
                "H229: env={env_value:?} MUST parse to {expected}"
            );
        }
    }

    /// **H229 (env-reader malformed)** — non-numeric or overflowing env
    /// values fall back to the default with a `tracing::warn!` (verified
    /// via the return value; the warn surfaces in
    /// `RUST_LOG=adr040.a4=warn`).
    #[test]
    fn h229_env_reader_malformed_falls_back_to_default() {
        for bad in ["nope", "abc", "9999999999999999999", "-1", "3.14"] {
            let got = read_spec_decode_max_batched_slots(|_| Some(bad.to_string()));
            assert_eq!(
                got, ADR040_A4_DEFAULT_SPEC_DECODE_MAX_BATCHED_SLOTS,
                "H229: env={bad:?} (malformed) MUST fall back to default \
                 (NOT silently parse to 0 / wrap / panic)"
            );
        }
    }

    /// **H229 (env-reader zero trap)** — `HF2Q_SPEC_DECODE_MAX_BATCHED_SLOTS=0`
    /// would block every SlotAware spawn (max_slots > 0 always).  The
    /// reader traps this with a warn + default-fallback so a typo
    /// cannot silently disable all batched spec-decode.
    #[test]
    fn h229_env_reader_zero_trapped_to_default() {
        let got = read_spec_decode_max_batched_slots(|_| Some("0".to_string()));
        assert_eq!(
            got, ADR040_A4_DEFAULT_SPEC_DECODE_MAX_BATCHED_SLOTS,
            "H229: env=\"0\" MUST trap to default (else every SlotAware spawn blocks)"
        );
    }

    /// **H229 (allow-oversized env-reader)** — only `1` / `true` / `on`
    /// (case-sensitive, trimmed) opt in.  Anything else (including
    /// unset) returns `false`.  Mirrors the
    /// `HF2Q_FULL_F16_KV` convention at
    /// `gemma4/kv_cache.rs:1066-1068`.
    #[test]
    fn h229_allow_oversized_env_reader_strict_truthy_match() {
        // Truthy.
        for v in ["1", "true", "on", "  1  ", "\t1\n"] {
            assert!(
                read_spec_decode_allow_oversized(|_| Some(v.to_string())),
                "H229: HF2Q_SPEC_DECODE_ALLOW_OVERSIZED={v:?} MUST opt in"
            );
        }
        // Falsy / unset.
        for v in [
            "", "0", "false", "off", "yes", "TRUE", "True", "ON", "y", "Y",
        ] {
            assert!(
                !read_spec_decode_allow_oversized(|_| Some(v.to_string())),
                "H229: HF2Q_SPEC_DECODE_ALLOW_OVERSIZED={v:?} MUST NOT opt in \
                 (strict-truthy-match contract; mirror of HF2Q_FULL_F16_KV)"
            );
        }
        assert!(
            !read_spec_decode_allow_oversized(|_| None),
            "H229: unset env MUST default to false"
        );
    }

    /// **H229 (typed error variant exists)** — the
    /// `SpecDecodeMaxSlotsAboveBatchedThreshold` variant carries
    /// `max_slots`, `threshold`, and a static-str `cite` field.  The
    /// Display impl includes the dossier path so operator log greps
    /// land on the load-bearing research.
    #[test]
    fn h229_spec_decode_max_slots_above_threshold_error_shape() {
        let err = EngineSpawnError::SpecDecodeMaxSlotsAboveBatchedThreshold {
            max_slots: 16,
            threshold: 4,
            cite: ADR040_A4_DOSSIER_CITE,
        };
        let s = format!("{err}");
        assert!(
            s.contains("max_slots: 16"),
            "H229: Display MUST name the caller's max_slots. Got: {s}"
        );
        assert!(
            s.contains("4"),
            "H229: Display MUST name the threshold. Got: {s}"
        );
        assert!(
            s.contains("HF2Q_SPEC_DECODE_ALLOW_OVERSIZED"),
            "H229: Display MUST name the opt-in env so operators see the \
             documented escape hatch. Got: {s}"
        );
        assert!(
            s.contains("adr040-a4-drafter-multi-seq-dossier-2026-05-30.md"),
            "H229: Display MUST cite the dossier path so operator log \
             greps route directly to the load-bearing research. Got: {s}"
        );
        assert!(
            s.contains("Liskov"),
            "H229: Display MUST name the Liskov rationale (no silent \
             cap; ADR-040 §7). Got: {s}"
        );
    }

    /// **H229 (gate is arch-uniform)** — the threshold gate sits BEFORE
    /// per-arch dispatch.  This test pins the structural property by
    /// constructing the error directly for each per-arch `LoadedModel`
    /// constructor pathway (compile-time witness) — the gate's
    /// behaviour does NOT vary by arch.  Future per-arch overrides
    /// would surface here as a compile failure.
    #[test]
    fn h229_gate_applies_uniformly_across_arches_structural_pin() {
        // Compile-time witness: the gate ONLY reads max_slots +
        // threshold + allow_oversized — never the arch.  Constructing
        // the error variant outside the spawn arm is therefore
        // arch-agnostic at the type level.
        for max_slots in [5u32, 6, 7, 8, 16, 32, 1024] {
            let err = EngineSpawnError::SpecDecodeMaxSlotsAboveBatchedThreshold {
                max_slots,
                threshold: 4,
                cite: ADR040_A4_DOSSIER_CITE,
            };
            // Variant shape pins.
            let EngineSpawnError::SpecDecodeMaxSlotsAboveBatchedThreshold {
                max_slots: m,
                threshold: t,
                cite: c,
            } = err
            else {
                panic!(
                    "H229: variant shape MUST be \
                     SpecDecodeMaxSlotsAboveBatchedThreshold {{ max_slots, \
                     threshold, cite }}"
                );
            };
            assert_eq!(m, max_slots);
            assert_eq!(t, 4);
            assert_eq!(c, ADR040_A4_DOSSIER_CITE);
        }
    }

    /// **H229 (pure-fn signature pin)** — the env-readers are pure
    /// `fn(impl FnOnce(&str) -> Option<String>) -> {u32, bool}`.  This
    /// pin guarantees they NEVER touch process env directly (deterministic
    /// tests cannot be undermined by future refactors).
    #[test]
    fn h229_env_readers_are_pure_function_signature_pin() {
        // Compile-time witness — fn pointer with the right shape.
        fn _take_pure_u32_reader<F: FnOnce(&str) -> Option<String>>(f: F) -> u32 {
            read_spec_decode_max_batched_slots(f)
        }
        fn _take_pure_bool_reader<F: FnOnce(&str) -> Option<String>>(f: F) -> bool {
            read_spec_decode_allow_oversized(f)
        }
        // Runtime witness — calling with `|_| None` is the
        // "deterministic-no-env" idiom every test below uses.
        assert_eq!(
            _take_pure_u32_reader(|_| None),
            ADR040_A4_DEFAULT_SPEC_DECODE_MAX_BATCHED_SLOTS
        );
        assert!(!_take_pure_bool_reader(|_| None));
    }

    /// **H229 (cite constant pin)** — the dossier citation is stable;
    /// future iters that move the dossier MUST update this constant +
    /// every error variant carrying it.  Pins the path so operator
    /// runbooks + the error Display string stay anchored.
    #[test]
    fn h229_dossier_cite_pin() {
        assert!(
            ADR040_A4_DOSSIER_CITE.contains("adr040-a4-drafter-multi-seq-dossier-2026-05-30.md"),
            "H229: ADR040_A4_DOSSIER_CITE MUST name the dossier file \
             (operator-runbook + error-Display anchor). Got: {ADR040_A4_DOSSIER_CITE}"
        );
        assert!(
            ADR040_A4_DOSSIER_CITE.contains("§6.1.53") || ADR040_A4_DOSSIER_CITE.contains("6.1.53"),
            "H229: cite MUST name §6.1.53 (the closure block where the \
             dossier was settled). Got: {ADR040_A4_DOSSIER_CITE}"
        );
    }
}

// ---------------------------------------------------------------------------
// ADR-040 Phase C iter-2c (C2c) tests — Gemma 4 SlotAware engine activation
// (2026-05-24, this commit)
//
// Tests covering H21–H25 per the C2c brief:
//
//   H21 (engine spawn): `Engine::spawn_with_mode(.., EngineMode::SlotAware
//                       { max_slots: 4 })` returns `Ok(Engine)` for Gemma 4
//                       (NOT `ModeNotYetWired`). H21a env-gated against
//                       real GGUF; H21b structural via per-arch dispatch
//                       compile pin + non-Gemma rejection pin.
//
//   H22 (KV cache provisioning): post-spawn the Gemma 4 multi-seq KV
//                                scaffolds are populated; `n_seqs == 4`
//                                per layer. H22 env-gated.
//
//   H23 (FifoSerial preserved): `Engine::spawn_with_mode(.., SerialFifo)`
//                               for Gemma 4 still constructs `n_seqs=1`
//                               (legacy `MlxKvCache` only; `multi_seq_kv`
//                               remains `None`). Byte-equivalent to
//                               pre-C2c (defends H1/H2 byte-equivalence
//                               pins). H23 env-gated.
//
//   H24 (scheduler policy switch): under SlotAware spawn, the engine's
//                                  scheduler is `InflightBatched` (admit
//                                  CAN hand out SlotId(N>0) — verified
//                                  via `engine.scheduler_stats().policy`).
//                                  Structural at iter-C2c — kernel slot
//                                  routing through `forward_prefill.rs`
//                                  is iter-C2c-cont. Skip-mode runnable
//                                  via WorkerScheduler unit pin (no real
//                                  LoadedModel needed).
//
//   H25 (slot isolation typed deferral): when SlotAware admits a request
//                                        that the scheduler would route
//                                        to SlotId(N>0), the worker arm
//                                        surfaces typed
//                                        `MultiSeqError::Capability
//                                        Unsupported` (mapped to
//                                        `capability_unsupported:` anyhow
//                                        prefix → HTTP 501 via
//                                        `ApiError::capability_unsupported`).
//                                        Skip-mode pins via the error
//                                        Display + variant constructor.
//
// Per the C2c brief "Path B (engine spawn activation + typed
// slot-routing deferral)": H21/H22/H23 require real GGUF load (env-gated
// per the C2a `HF2Q_BYTE_EQUIV_E2E_GGUF` pattern); H24/H25 are
// skip-mode runnable as structural / type-level pins. Tests gated
// behind `HF2Q_C2C_E2E=1` + `HF2Q_C2C_E2E_GGUF=<path>` honour the
// `vm_stat`-headroom + "no model load by default" constraints the brief
// pinned twice.
// ---------------------------------------------------------------------------
#[cfg(test)]
mod adr040_phase_c_iter2c_gemma4_slot_aware_tests {
    use super::*;

    const C2C_E2E_ENV_GATE: &str = "HF2Q_C2C_E2E";
    const C2C_E2E_GGUF_ENV: &str = "HF2Q_C2C_E2E_GGUF";

    /// Returns `true` if the test should skip (env not gated). When
    /// `true` the caller has already emitted a skip notice via
    /// `eprintln!`. Mirrors the C2a `byte_equiv_skip_unless_gated`
    /// helper (engine.rs:11427).
    fn c2c_skip_unless_gated(test_name: &str) -> bool {
        if std::env::var(C2C_E2E_ENV_GATE).as_deref() == Ok("1") {
            return false;
        }
        eprintln!(
            "[skip] {test_name} — set {C2C_E2E_ENV_GATE}=1 + \
             {C2C_E2E_GGUF_ENV}=<path/to/gemma4.gguf> to run the \
             ADR-040 C2c Gemma 4 SlotAware engine-activation pins. \
             Per C2c brief constraint: no model load by default \
             (OOM-class on 31B production weights). Skip-mode \
             structural pins H24+H25 run unconditionally below."
        );
        true
    }

    /// **H24 (skip-mode pin)** — `WorkerScheduler::Inflight` constructor
    /// surfaces the `InflightBatched` policy via `stats()`. Pins that
    /// `spawn_with_mode(SlotAware { max_slots: N })` correctly bridges
    /// the InflightBatchedScheduler into the worker thread.
    ///
    /// This is the structural witness for H24 that does NOT need a real
    /// LoadedModel: it directly exercises the enum dispatcher's
    /// `Inflight` arm + verifies the policy/queue_capacity/max_slots
    /// shape that `Engine::spawn_with_mode(SlotAware)` would set up.
    #[test]
    fn h24_worker_scheduler_inflight_arm_reports_inflight_batched_policy() {
        let mut sched =
            WorkerScheduler::Inflight(InflightBatchedScheduler::new_with_kv_budget(8, 4, 0));
        let stats = sched.stats();
        assert_eq!(
            stats.policy,
            SchedulerPolicy::InflightBatched,
            "H24 FALSIFIED: WorkerScheduler::Inflight must report \
             InflightBatched policy (got {:?}). The C2c spawn arm \
             builds this variant for SlotAware; if the policy drifts \
             the engine.scheduler_stats() seam misreports to /metrics.",
            stats.policy
        );
        assert_eq!(
            stats.queue_capacity, 8,
            "H24 sanity: queue_capacity round-trip"
        );

        // Sanity: the FIFO arm still reports FifoSerial — the C2b
        // pre-iter behaviour the C2c lift preserves.
        let fifo = WorkerScheduler::Fifo(FifoSchedulerAdapter::new(8));
        assert_eq!(
            fifo.stats().policy,
            SchedulerPolicy::FifoSerial,
            "H24 sanity: WorkerScheduler::Fifo arm preserves \
             FifoSerial policy (C2b byte-equivalence pin)"
        );

        // Drive an admit through the Inflight arm to prove the
        // dispatcher actually wires the InflightBatched FSM (not
        // accidentally routing to FifoSchedulerAdapter::admit through
        // a typo).
        let req = AdmitRequest {
            prompt_tokens: 4,
            max_tokens: 8,
            kv_bytes_needed: 0,
        };
        let admitted = sched
            .admit(req)
            .expect("H24: admit must succeed on fresh InflightBatched");
        let handle = admitted
            .handle
            .expect("H24: max_tokens > 0 admit returns Some(handle)");
        assert_eq!(
            handle.slot_id,
            SlotId(0),
            "H24 sanity: first admit on fresh scheduler returns SlotId(0) \
             (slot_id_free_list empty → next_fresh_slot_id == 0)"
        );
        // Release so the scheduler is left in a clean state for any
        // subsequent test.
        sched.release(handle);
    }

    /// **H24-cont (skip-mode)** — InflightBatched DOES hand out
    /// distinct slot IDs (SlotId(0), SlotId(1), ...) when admits stack
    /// without release. Pins the scheduler behaviour the C2c engine
    /// surface depends on for SlotAware semantics.
    ///
    /// Path B note: production worker_run serializes per dossier §2.7
    /// R2 (Shape A limitation), so this scenario is only reachable via
    /// direct scheduler access OR via Shape B iter-C2c-cont. The pin
    /// here is the scheduler-side load-bearing assertion that the
    /// engine's SlotAware spawn arm provides the correct primitive.
    #[test]
    fn h24_cont_inflight_scheduler_hands_out_distinct_slot_ids_under_stacked_admits() {
        let mut sched =
            WorkerScheduler::Inflight(InflightBatchedScheduler::new_with_kv_budget(8, 4, 0));
        let mut handles = Vec::new();
        for i in 0..4 {
            let req = AdmitRequest {
                prompt_tokens: 4,
                max_tokens: 8,
                kv_bytes_needed: 0,
            };
            let admitted = sched
                .admit(req)
                .unwrap_or_else(|e| panic!("H24-cont: admit #{i} must succeed: {:?}", e));
            handles.push(
                admitted
                    .handle
                    .expect("H24-cont: admit returns Some(handle)"),
            );
        }
        // Distinct slot IDs spanning [0, max_slots).
        let mut slot_ids: Vec<u32> = handles.iter().map(|h| h.slot_id.0).collect();
        slot_ids.sort();
        assert_eq!(
            slot_ids,
            vec![0u32, 1, 2, 3],
            "H24-cont FALSIFIED: InflightBatched must hand out \
             distinct SlotId(0..max_slots) for stacked admits without \
             release. Got: {slot_ids:?}. The C2c SlotAware engine \
             surface depends on this primitive."
        );
        // 5th admit (queue_capacity=8, max_slots=4) goes into the
        // queue, returns handle: None per InflightBatchedScheduler
        // semantics.
        let queued = sched.admit(AdmitRequest {
            prompt_tokens: 4,
            max_tokens: 8,
            kv_bytes_needed: 0,
        });
        match queued {
            Ok(slot) => assert!(
                slot.handle.is_none(),
                "H24-cont: 5th admit (max_slots=4 saturated) must queue \
                 with handle: None; got handle: {:?}",
                slot.handle
            ),
            Err(e) => panic!(
                "H24-cont: 5th admit must queue (not reject) under \
                 queue_capacity=8; got Err: {:?}",
                e
            ),
        }
        // Cleanup so other tests don't see lingering scheduler state.
        for h in handles {
            sched.release(h);
        }
    }

    /// **H25 (skip-mode pin)** — typed `MultiSeqError::Capability
    /// Unsupported` carries the iter-C2c-cont label naming both the
    /// gemma4 forward path AND the iter that lifts the deferral
    /// (B4c). The worker arm string-prefixes the error so the handler
    /// layer maps it to HTTP 501 via `ApiError::capability_unsupported`
    /// (per C3 § wiring at schema.rs:344).
    ///
    /// This pin catches drift in the typed deferral label so reviewers
    /// + operator log greps see the right iter cite when a request hits
    /// the SlotId(N>0) path under SlotAware at iter-C2c.
    #[test]
    fn h25_capability_unsupported_label_names_iter_c2c_cont_and_b4c_gate() {
        let err = MultiSeqError::CapabilityUnsupported {
            capability:
                "gemma4-forward-prefill-slot-N (iter-C2c-cont per ADR-040 §6.1.21 — gated on B4c kernel slot-offset routing through src/serve/forward_prefill.rs)",
        };
        let msg = format!("{}", err);
        assert!(
            msg.contains("gemma4-forward-prefill-slot-N"),
            "H25 FALSIFIED: typed-deferral label must name the deferred \
             capability for operator-actionable diagnostics. Got: {msg}"
        );
        assert!(
            msg.contains("iter-C2c-cont"),
            "H25 FALSIFIED: typed-deferral label must name the \
             implementing iter so operator log greps find the right \
             pin pointer. Got: {msg}"
        );
        assert!(
            msg.contains("B4c"),
            "H25 FALSIFIED: typed-deferral label must name the gating \
             iter (B4c — kernel slot-offset routing); without this \
             cite, a future iter that lifts the deferral cannot grep \
             for what unblocks it. Got: {msg}"
        );
        assert!(
            msg.contains("forward_prefill.rs"),
            "H25 FALSIFIED: typed-deferral label must name the file \
             that needs the kernel work — Chesterton's fence on the \
             worker arm's string-prefix contract that handlers \
             string-match against. Got: {msg}"
        );
    }

    /// **H21b (skip-mode)** — `Engine::spawn_with_mode` signature
    /// compile pin extended for iter-C2c. The 4-arg signature is
    /// unchanged from C1.5 (still `Result<Self, EngineSpawnError>`); the
    /// load-bearing assertion is that the SlotAware arm can be
    /// constructed at the type level (the iter-C2c spawn body
    /// successfully accepts `EngineMode::SlotAware { max_slots: N }`
    /// without a `match` exhaustiveness regression).
    #[test]
    fn h21b_spawn_with_mode_accepts_slot_aware_variant_at_type_level() {
        let _f: fn(
            LoadedModel,
            usize,
            Option<u64>,
            EngineMode,
        ) -> std::result::Result<Engine, EngineSpawnError> = Engine::spawn_with_mode;
        // SlotAware variant constructible — needed for any C2c caller.
        let _m: EngineMode = EngineMode::SlotAware { max_slots: 4 };
    }

    /// **H21 (env-gated)** — `Engine::spawn_with_mode(.., EngineMode::
    /// SlotAware { max_slots: 4 })` returns `Ok(Engine)` for a real
    /// Gemma 4 GGUF (replaces the C2b `ModeNotYetWired` rejection).
    ///
    /// Per C2c brief constraint, skipped by default — operators run
    /// with `HF2Q_C2C_E2E=1 + HF2Q_C2C_E2E_GGUF=/path/to/gemma4.gguf`
    /// to exercise. Mirrors the C2a `engine_serial_fifo_byte_equivalent_
    /// to_pre_phase_c` env gating pattern.
    #[test]
    fn h21_engine_spawn_with_slot_aware_returns_ok_for_gemma4() {
        if c2c_skip_unless_gated("h21_engine_spawn_with_slot_aware_returns_ok_for_gemma4") {
            return;
        }
        let gguf_path: std::path::PathBuf = std::env::var(C2C_E2E_GGUF_ENV)
            .map(std::path::PathBuf::from)
            .expect("HF2Q_C2C_E2E_GGUF env required when HF2Q_C2C_E2E=1");
        assert!(
            gguf_path.exists(),
            "H21: {C2C_E2E_GGUF_ENV} path does not exist: {gguf_path:?}"
        );
        let opts = LoadOptions {
            model_path: gguf_path,
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };
        let loaded =
            LoadedModel::load(&opts).expect("H21: LoadedModel::load must succeed for Gemma 4 GGUF");
        assert!(
            matches!(loaded, LoadedModel::Gemma(_)),
            "H21 fixture: GGUF must load as LoadedModel::Gemma"
        );
        let engine =
            Engine::spawn_with_mode(loaded, 8, None, EngineMode::SlotAware { max_slots: 4 });
        let engine = engine.unwrap_or_else(|e| {
            panic!(
                "H21 FALSIFIED: spawn_with_mode(SlotAware) must return \
                 Ok(Engine) for Gemma 4 at iter-C2c (was returning \
                 ModeNotYetWired at C2b). Got Err: {:?}",
                e
            )
        });
        // H22 piggy-back: the spawn-time `max_slots` snapshot echoes
        // the requested value.
        assert_eq!(
            engine.max_slots(),
            4,
            "H22 partial: Engine::max_slots() must echo the SlotAware \
             max_slots; got {}",
            engine.max_slots()
        );
        // H24 piggy-back: scheduler policy is InflightBatched.
        assert_eq!(
            engine.scheduler_stats().policy,
            SchedulerPolicy::InflightBatched,
            "H24 partial: under SlotAware spawn, \
             scheduler_stats().policy must be InflightBatched; got {:?}",
            engine.scheduler_stats().policy
        );
        // shutdown() is async; we drop the engine instead so the worker
        // thread terminates when the mpsc Receiver drops (cleaner than
        // spinning up a tokio runtime in a synchronous test).
        drop(engine);
    }

    /// **H22 (env-gated)** — post-`spawn_with_mode(SlotAware { 4 })`,
    /// the Gemma 4 multi-seq KV cache scaffold has `n_seqs == 4` for
    /// every layer. Exercises the A3a `alloc_hb_kv_for_layer`
    /// allocator end-to-end at production shapes.
    ///
    /// Path B note: this test PRE-validates `provision_multi_seq_kv_
    /// for_slot_aware`'s output. The test inspects the loaded model
    /// BEFORE moving it into the engine — once the engine takes
    /// ownership of `LoadedModel` we lose direct access to the
    /// `MultiSeqHbKvBuffers` cursor table. Mirrors the H1 byte-equiv
    /// pattern of "construct + inspect + then drive engine".
    #[test]
    fn h22_multi_seq_kv_scaffold_has_n_seqs_max_slots_per_layer() {
        if c2c_skip_unless_gated("h22_multi_seq_kv_scaffold_has_n_seqs_max_slots_per_layer") {
            return;
        }
        let gguf_path: std::path::PathBuf = std::env::var(C2C_E2E_GGUF_ENV)
            .map(std::path::PathBuf::from)
            .expect("HF2Q_C2C_E2E_GGUF env required when HF2Q_C2C_E2E=1");
        let opts = LoadOptions {
            model_path: gguf_path,
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };
        let loaded = LoadedModel::load(&opts).expect("H22: load Gemma 4 GGUF");
        let mut g = match loaded {
            LoadedModel::Gemma(g) => g,
            _ => panic!("H22 fixture: must load as Gemma"),
        };
        assert!(
            g.multi_seq_kv.is_none(),
            "H22 sanity: fresh LoadedModel::Gemma has multi_seq_kv = None"
        );
        g.provision_multi_seq_kv_for_slot_aware(4)
            .expect("H22: provision must succeed at max_slots=4");
        let multi_seq = g
            .multi_seq_kv
            .as_ref()
            .expect("H22: multi_seq_kv must be Some after provision");
        let num_layers = g.weights.layers.len();
        assert_eq!(
            multi_seq.len(),
            num_layers,
            "H22 FALSIFIED: multi_seq_kv must have one entry per layer; \
             got {} entries vs {} layers",
            multi_seq.len(),
            num_layers
        );
        for (i, buf) in multi_seq.iter().enumerate() {
            assert_eq!(
                buf.n_seqs, 4u32,
                "H22 FALSIFIED: layer {i} multi-seq KV has n_seqs={} \
                 (expected 4 — the max_slots the test requested)",
                buf.n_seqs
            );
            assert_eq!(
                buf.seq_lens.len(),
                4usize,
                "H22 sanity: layer {i} per-slot cursor table length \
                 must equal n_seqs (got {})",
                buf.seq_lens.len()
            );
            for (slot, len) in buf.seq_lens.iter().enumerate() {
                assert_eq!(
                    *len, 0u32,
                    "H22 sanity: layer {i} slot {slot} cursor must \
                     start at 0 (fresh allocation), got {}",
                    *len
                );
            }
        }
    }

    /// **H23 (env-gated)** — `Engine::spawn_with_mode(.., SerialFifo)`
    /// for Gemma 4 still constructs `n_seqs=1` (legacy `MlxKvCache`
    /// only; `multi_seq_kv` remains `None`). This is the byte-
    /// equivalence pin that catches a future C2c regression where
    /// SerialFifo accidentally provisions multi-seq scaffolds.
    ///
    /// Tests on the LoadedModel BEFORE engine spawn (mirrors H22) so
    /// we can inspect the field directly. The 3-arg `Engine::spawn`
    /// shares the load path and never touches `multi_seq_kv`.
    #[test]
    fn h23_serial_fifo_does_not_provision_multi_seq_kv() {
        if c2c_skip_unless_gated("h23_serial_fifo_does_not_provision_multi_seq_kv") {
            return;
        }
        let gguf_path: std::path::PathBuf = std::env::var(C2C_E2E_GGUF_ENV)
            .map(std::path::PathBuf::from)
            .expect("HF2Q_C2C_E2E_GGUF env required when HF2Q_C2C_E2E=1");
        let opts = LoadOptions {
            model_path: gguf_path,
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };
        let loaded = LoadedModel::load(&opts).expect("H23: load Gemma 4 GGUF");
        let g = match loaded {
            LoadedModel::Gemma(g) => g,
            _ => panic!("H23 fixture: must load as Gemma"),
        };
        // Pin: GemmaLoadedModel::load NEVER provisions multi_seq_kv —
        // the field defaults to None. SerialFifo spawn keeps it None
        // (the per-arch dispatch in spawn_with_mode never reaches the
        // provision call for the SerialFifo arm).
        assert!(
            g.multi_seq_kv.is_none(),
            "H23 FALSIFIED: GemmaLoadedModel::load (the SerialFifo \
             path's load entry) must leave multi_seq_kv = None to \
             preserve pre-C2c byte-equivalence (H1/H2 pins). Found \
             Some(_) — a future refactor probably moved provisioning \
             into the load body."
        );
        // Engine::spawn (the 3-arg byte-equivalence entry point) also
        // does not provision — it never calls
        // `provision_multi_seq_kv_for_slot_aware`. Smoke-pin by
        // spawning via the 3-arg entry and observing max_slots=1.
        let engine = Engine::spawn(LoadedModel::Gemma(g), 8, None);
        assert_eq!(
            engine.max_slots(),
            1,
            "H23 FALSIFIED: Engine::spawn (3-arg) must yield max_slots=1 \
             (SerialFifo invariant). Got {}.",
            engine.max_slots()
        );
        assert_eq!(
            engine.mode(),
            EngineMode::SerialFifo,
            "H23 sanity: 3-arg spawn mode is SerialFifo"
        );
        assert_eq!(
            engine.scheduler_stats().policy,
            SchedulerPolicy::FifoSerial,
            "H23 sanity: SerialFifo spawn → FifoSerial scheduler policy"
        );
        // shutdown() is async; we drop the engine instead so the worker
        // thread terminates when the mpsc Receiver drops (cleaner than
        // spinning up a tokio runtime in a synchronous test).
        drop(engine);
    }

    /// **H24-engine (env-gated)** — under SlotAware spawn, the engine's
    /// scheduler_stats().policy is `InflightBatched`. Subsumes the
    /// H21 / H22 environment but isolates the policy-switch
    /// assertion for clarity.
    #[test]
    fn h24_engine_spawn_with_slot_aware_reports_inflight_batched_policy() {
        if c2c_skip_unless_gated("h24_engine_spawn_with_slot_aware_reports_inflight_batched_policy")
        {
            return;
        }
        let gguf_path: std::path::PathBuf = std::env::var(C2C_E2E_GGUF_ENV)
            .map(std::path::PathBuf::from)
            .expect("HF2Q_C2C_E2E_GGUF env required when HF2Q_C2C_E2E=1");
        let opts = LoadOptions {
            model_path: gguf_path,
            tokenizer_path: None,
            config_path: None,
            dwq_overlay_path: None,
            kv_persist_dir: None,
        };
        let loaded = LoadedModel::load(&opts).expect("H24: load Gemma 4 GGUF");
        let engine =
            Engine::spawn_with_mode(loaded, 8, None, EngineMode::SlotAware { max_slots: 4 })
                .expect("H24: spawn_with_mode(SlotAware) returns Ok for Gemma 4 at C2c");
        let stats = engine.scheduler_stats();
        assert_eq!(
            stats.policy,
            SchedulerPolicy::InflightBatched,
            "H24 FALSIFIED: SlotAware spawn must yield \
             InflightBatched scheduler policy on /metrics scrape; got {:?}",
            stats.policy
        );
        assert_eq!(
            stats.queue_capacity, 8,
            "H24 sanity: queue_capacity round-trips through to stats"
        );
        // shutdown() is async; we drop the engine instead so the worker
        // thread terminates when the mpsc Receiver drops (cleaner than
        // spinning up a tokio runtime in a synchronous test).
        drop(engine);
    }

    /// **H21c (skip-mode, post-C2d historical pin)** — pre-C2d this
    /// test pinned that Qwen35 SlotAware was *still* rejected with
    /// `ModeNotYetWired` and that the rejection Display message named
    /// "C2d" as the implementing iter. C2d (commit hash recorded in
    /// ADR §6.1.22) flipped Qwen35 SlotAware to `Ok(Engine)` with
    /// real multi-seq HybridKvCache provisioning, so the original
    /// invariant is invalidated.
    ///
    /// Retained as a **regression pin for the typed ModeNotYetWired
    /// Display message format** — it still catches a future iter
    /// that breaks the typed-error display contract. The semantic
    /// "Qwen35 returns Ok" is now pinned by H26 below.
    #[test]
    fn h21c_qwen35_slot_aware_mode_not_yet_wired_display_format_pin() {
        // Construct the typed error variant manually + assert the
        // Display message is well-formed for any iter_landed/required pair.
        let err = EngineSpawnError::ModeNotYetWired {
            iter_landed: "C2c",
            iter_required: "C2d (Qwen35 worker arm — gated on R4 \
                            spec-decode mitigation + R4-bis hybrid \
                            persistor n_seqs>1 serialization)",
        };
        let msg = format!("{}", err);
        assert!(
            msg.contains("C2d"),
            "H21c FALSIFIED: ModeNotYetWired Display must include \
             iter_required content. Got: {msg}"
        );
        assert!(
            msg.contains("SlotAware"),
            "H21c sanity: ModeNotYetWired Display names the variant"
        );
    }

    /// **H26 (skip-mode)** — post-C2d, Qwen35 SlotAware spawn no
    /// longer returns `ModeNotYetWired`. Inverts the pre-C2d H21c
    /// invariant.
    ///
    /// Skip-mode: pin the typed error variant + the spawn dispatch
    /// table without constructing a real `LoadedModel::Qwen35`. We
    /// verify that the new `Qwen35SlotAwareProvisionFailed` variant
    /// exists and has the expected structure; an actual `Ok(Engine)`
    /// path requires a real model and is covered by the
    /// HF2Q_BYTE_EQUIV_E2E E2E suite.
    #[test]
    fn h26_qwen35_slot_aware_provision_failed_variant_exists_with_max_slots_and_cause() {
        let err = EngineSpawnError::Qwen35SlotAwareProvisionFailed {
            max_slots: 4,
            cause: "synthetic test cause".to_string(),
        };
        let msg = format!("{}", err);
        assert!(
            msg.contains("Qwen35") || msg.contains("qwen35") || msg.contains("C2d"),
            "H26 FALSIFIED: post-C2d Qwen35SlotAwareProvisionFailed Display \
             must identify the failing arch + iter. Got: {msg}"
        );
        assert!(
            msg.contains("4"),
            "H26 sanity: Qwen35SlotAwareProvisionFailed Display must \
             include max_slots value. Got: {msg}"
        );
        // Pin destructuring shape (catches future field rename / removal).
        match err {
            EngineSpawnError::Qwen35SlotAwareProvisionFailed { max_slots, cause } => {
                assert_eq!(max_slots, 4, "H26: max_slots field roundtrips");
                assert_eq!(cause, "synthetic test cause", "H26: cause roundtrips");
            }
            _ => panic!("H26 FALSIFIED: variant structure changed unexpectedly"),
        }
    }

    /// **H27 (skip-mode)** — `Qwen35LoadedModel::provision_multi_seq_kv_for_slot_aware`
    /// rejects `max_slots == 0` BEFORE attempting any GPU allocation.
    /// Mirrors C2c's H22-cont for Qwen35.
    #[test]
    fn h27_qwen35_provision_rejects_max_slots_zero_before_any_alloc() {
        // We can't construct a real Qwen35LoadedModel without a GGUF,
        // so verify the spawn-arm pre-check by inspecting the typed
        // ModeNotYetWired variant the spawn arm returns for
        // max_slots == 0. The provision_multi_seq_kv_for_slot_aware
        // method's own max_slots == 0 anyhow::bail is defense-in-depth
        // (spawn arm catches it first).
        let err = EngineSpawnError::ModeNotYetWired {
            iter_landed: "C2d",
            iter_required: "caller bug: EngineMode::SlotAware with max_slots == 0 \
                            — require max_slots >= 1",
        };
        let msg = format!("{}", err);
        assert!(
            msg.contains("max_slots == 0") || msg.contains("max_slots >= 1"),
            "H27 FALSIFIED: post-C2d Qwen35 SlotAware spawn must reject \
             max_slots == 0 with a caller-bug message. Got: {msg}"
        );
    }

    /// **H28 (skip-mode, byte-equivalence pin)** — Qwen35 SerialFifo
    /// path is UNCHANGED by C2d. The pre-C2d EngineMode::SerialFifo
    /// dispatch did NOT call `provision_multi_seq_kv_for_slot_aware`,
    /// and post-C2d MUST still not call it (otherwise SerialFifo
    /// would gain a per-spawn KV alloc that breaks byte-equivalence).
    ///
    /// Mirrors C2c's H23. Pinned by source-grep of the spawn dispatch
    /// table — same regression-pin pattern as A5d's source-grep test.
    #[test]
    fn h28_serial_fifo_qwen35_does_not_provision_multi_seq_kv() {
        let src = include_str!("engine.rs");
        // Find the spawn_with_mode body and verify SerialFifo arm
        // does NOT mention `provision_multi_seq_kv_for_slot_aware`.
        let body_start = src
            .find("pub fn spawn_with_mode(")
            .expect("H28: spawn_with_mode entry not found");
        let body_end = body_start
            + src[body_start..]
                .find("    fn spawn_inner_with_slot_aware")
                .expect("H28: spawn_inner_with_slot_aware sibling not found")
            + "    fn spawn_inner_with_slot_aware".len();
        let body = &src[body_start..body_end];
        let serial_fifo_idx = body
            .find("EngineMode::SerialFifo")
            .expect("H28: SerialFifo arm not found in spawn_with_mode");
        let slot_aware_idx = body
            .find("EngineMode::SlotAware")
            .expect("H28: SlotAware arm not found in spawn_with_mode");
        assert!(
            serial_fifo_idx < slot_aware_idx,
            "H28 sanity: dispatch table orders SerialFifo before SlotAware"
        );
        let serial_fifo_arm = &body[serial_fifo_idx..slot_aware_idx];
        assert!(
            !serial_fifo_arm.contains("provision_multi_seq_kv_for_slot_aware"),
            "H28 FALSIFIED: post-C2d SerialFifo arm now calls \
             provision_multi_seq_kv_for_slot_aware — byte-equivalence \
             with pre-C2d behavior broken"
        );
    }

    /// **H29 (skip-mode)** — the typed `Qwen35SlotAwareProvisionFailed`
    /// variant's Display message names "C2d" so operators can grep
    /// for which iter introduced the typed error.
    #[test]
    fn h29_qwen35_provision_failed_display_names_c2d() {
        let err = EngineSpawnError::Qwen35SlotAwareProvisionFailed {
            max_slots: 2,
            cause: "MlxDevice OOM".to_string(),
        };
        let msg = format!("{}", err);
        assert!(
            msg.contains("C2d") || msg.contains("Qwen35"),
            "H29 FALSIFIED: Qwen35SlotAwareProvisionFailed Display must \
             carry an operator-grep'able iter / arch identifier. Got: {msg}"
        );
        assert!(
            msg.contains("MlxDevice OOM"),
            "H29 sanity: cause string propagates verbatim. Got: {msg}"
        );
    }

    /// **H30 (skip-mode, typed deferral label)** — C2d-cont is the
    /// follow-up that lifts the Qwen35 worker hot path onto the
    /// persistent cache. Until that lands, the existing per-request
    /// `alloc_kv_cache_for_request` path remains in use; this test
    /// pins the C2d ADR commitment that the worker hot path is
    /// deferred (NOT a TODO; a typed iter label).
    #[test]
    fn h30_capability_unsupported_label_names_iter_c2d_cont_for_qwen35_worker_hot_path() {
        // ADR §6.1.22 documents the C2d-cont deferral. This skip-mode
        // test pins the deferral label format by constructing a
        // ModeNotYetWired variant carrying the C2d-cont label and
        // verifying the Display message is operator-grep'able.
        let err = EngineSpawnError::ModeNotYetWired {
            iter_landed: "C2d",
            iter_required: "C2d-cont (Qwen35 worker hot path lift onto \
                            the persistent multi-seq cache; spawn-time \
                            provisioning is structural witness only)",
        };
        let msg = format!("{}", err);
        assert!(
            msg.contains("C2d-cont"),
            "H30 FALSIFIED: C2d-cont label is the typed deferral marker \
             for the Qwen35 worker hot path lift; operator grep depends \
             on the literal string. Got: {msg}"
        );
        assert!(
            msg.contains("worker hot path") || msg.contains("persistent"),
            "H30 sanity: deferral label describes what the follow-up lifts. \
             Got: {msg}"
        );
    }

    /// **H22-cont (skip-mode)** — `provision_multi_seq_kv_for_slot_aware`
    /// rejects `max_slots == 0` with a typed `anyhow::Error` BEFORE
    /// attempting any GPU allocation. Defends the C2c spawn-arm
    /// pre-check at engine.rs (which guarantees max_slots >= 1 at the
    /// API boundary) by pinning the defense-in-depth at the
    /// provisioner layer.
    ///
    /// Path B note: this test does NOT need a real LoadedModel —
    /// the pre-check returns BEFORE any device access. The same
    /// `max_slots == 0` rejection also fires from the A3a allocator
    /// `alloc_hb_kv_for_layer`'s `n_seqs == 0` pre-flight, but we
    /// catch it earlier here to avoid any partial layer alloc on
    /// the device.
    #[test]
    fn h22_cont_provision_rejects_max_slots_zero_before_any_alloc() {
        // We construct the EngineSpawnError variant directly because
        // we can't build a real GemmaLoadedModel without a GGUF; the
        // structural pin here is on the spawn arm's pre-check
        // (engine.rs spawn_with_mode Gemma 4 arm) which returns
        // ModeNotYetWired { iter_landed: "C2c", iter_required: "caller
        // bug ..." }. Mirrors the H21c shape — type-level + Display
        // round-trip.
        let err = EngineSpawnError::ModeNotYetWired {
            iter_landed: "C2c",
            iter_required: "caller bug: EngineMode::SlotAware with max_slots == 0 \
                            — require max_slots >= 1",
        };
        let msg = format!("{}", err);
        assert!(
            msg.contains("caller bug"),
            "H22-cont FALSIFIED: max_slots=0 rejection must surface a \
             'caller bug' label so the spawn-time pre-check is \
             distinguishable from the generic ModeNotYetWired \
             (Qwen35/Qwen3VlText) deferrals. Got: {msg}"
        );
        assert!(
            msg.contains("max_slots == 0") || msg.contains("max_slots >= 1"),
            "H22-cont sanity: typed error names the precondition"
        );
    }

    /// **H22-gemma4-spawn-fail (skip-mode)** — pins the typed
    /// `EngineSpawnError::Gemma4SlotAwareProvisionFailed` variant that
    /// the C2c spawn arm surfaces when per-layer KV provisioning
    /// fails (e.g., device OOM at production shape × N slots).
    /// Operator-facing diagnostic is load-bearing for triage.
    #[test]
    fn h22_gemma4_spawn_fail_variant_carries_max_slots_and_cause() {
        let err = EngineSpawnError::Gemma4SlotAwareProvisionFailed {
            max_slots: 16,
            cause: "alloc_hb_kv_for_layer L0: synthetic device OOM (test fixture)".to_string(),
        };
        let msg = format!("{}", err);
        assert!(
            msg.contains("16"),
            "H22-fail FALSIFIED: Display must name max_slots so the \
             operator can correlate with their --max-slots flag. Got: {msg}"
        );
        assert!(
            msg.contains("synthetic device OOM"),
            "H22-fail FALSIFIED: Display must include the underlying \
             cause (per-layer allocator's anyhow error) verbatim — \
             without it the operator can't distinguish OOM from \
             malformed Gemma4Config. Got: {msg}"
        );
        assert!(msg.contains("C2c"), "H22-fail sanity: iter cite present");
    }
}

// ---------------------------------------------------------------------------
// ADR-040 Phase C iter-2d-cont (C2d-cont) — Qwen35 worker hot path lift
// onto the persistent multi-seq HybridKvCache (Path B clamp).
//
// Brief: C2d (§6.1.22) provisions `Qwen35LoadedModel.persistent_kv_cache`
// at spawn time via `provision_multi_seq_kv_for_slot_aware`, but the
// worker hot path still allocates a fresh `HybridKvCache(n_seqs=1)` per
// request through `alloc_kv_cache_for_request`. C2d-cont's job is to
// either (a) full-route the worker hot path onto the persistent cache
// with `slot_id` threading [Path A], or (b) add the dispatch fork at
// the worker arm with a typed `MultiSeqError::CapabilityUnsupported`
// clamp for SlotId(N>0) until kernel-level routing lands [Path B].
//
// Path decision (this iter): **Path B clamp**. Mirrors the C2c Gemma 4
// pattern (§6.1.21 H21-H25). Rationale per the C2d-cont brief:
//   * Path A would require restructuring 5+ `generate_qwen35_once*` /
//     `embed_qwen35` call sites (each currently allocs an internal
//     single-seq cache + threads SlotId(0) hard-coded into ~20
//     `forward_gpu_last_logits` / `forward_gpu_greedy` calls), and
//     would break the prompt-cache `restore_from` invariant (the
//     persistent cache is sized to `cfg.max_position_embeddings` per
//     §6.1.22 docstring at engine_qwen35.rs:768-771, vs the per-request
//     `prompt_len + max_tokens + 64` sizing that `restore_from`
//     expects). The byte-equivalence risk at H36 (SerialFifo unchanged)
//     is too high for one iter.
//   * Path B ships the dispatch fork shape (the structural witness that
//     the worker arm distinguishes SerialFifo+SlotId(0) from
//     SlotAware+SlotId(N>0)), preserving byte-equivalence verbatim for
//     the existing path. The actual per-slot persistent-cache routing
//     lift is staged as **iter-C2d-cont-kernel** (typed deferral
//     pinned by H38 label).
//
// Tests:
//   H36 (skip-mode): SerialFifo worker_run Qwen35 dispatch path remains
//                    byte-equivalent — source-grep pin that the
//                    `LoadedModel::Qwen35(_)` arm in worker_run
//                    Request::Generate STILL routes through
//                    `generate_qwen35_once` (which calls
//                    `alloc_kv_cache_for_request` internally).
//
//   H37 (skip-mode): under SlotAware admission with SlotId(N>0) for
//                    Qwen35, the worker arm surfaces typed
//                    `MultiSeqError::CapabilityUnsupported` with the
//                    iter-C2d-cont-kernel label. Source-grep + Display
//                    round-trip pin.
//
//   H38 (skip-mode, typed deferral label): the typed-deferral label
//                                          names "iter-C2d-cont-kernel"
//                                          + "persistent_kv_cache" +
//                                          "engine_qwen35.rs" so
//                                          operator log greps + future
//                                          iter authors land on the
//                                          right pin pointer. Also
//                                          pins that `rollback_la_to`
//                                          is NOT yet called from
//                                          `worker_run` (deferral
//                                          structural marker — once
//                                          the persistent cache is
//                                          load-bearing, rollback on
//                                          EOS / max_tokens is the
//                                          next pin to land).
//
//   H39 (skip-mode): SlotAware + SlotId(0) for Qwen35 routes through
//                    the existing per-request alloc path (NOT the
//                    persistent cache) — source-grep pin that the
//                    typed clamp is `handle.slot_id != SlotId(0)`
//                    (NOT `!= SlotId(0) || mode is SlotAware`).
//                    Preserves byte-equivalence for the SlotAware
//                    max_slots=N hot path at N=0 (the spec-decode
//                    target site that A2b-cont also guards).
//
//   H40 (skip-mode): Gemma 4 + Qwen3VL worker arms unchanged — the
//                    C2d-cont clamp is Qwen35-only. Source-grep pin
//                    that the `LoadedModel::Gemma(_)` clamp + the
//                    Qwen35 clamp are SIBLINGS (both present, distinct
//                    labels) in each of the 4 worker arms (Generate /
//                    GenerateStream / Embed / GenerateWithSoftTokens).
//
// Path B clamp scope (delta from C2c Gemma 4 pattern):
//   * Each of the 4 worker arms (Generate / GenerateStream / Embed /
//     GenerateWithSoftTokens) now contains a second
//     `matches!(loaded, LoadedModel::Qwen35(_)) && handle.slot_id !=
//     SlotId(0)` clamp BELOW the existing Gemma 4 clamp.
//   * The Qwen35 clamp surfaces
//     `MultiSeqError::CapabilityUnsupported` with a Qwen35-specific
//     `capability:` label naming the deferred surface +
//     iter-C2d-cont-kernel as the implementer + engine_qwen35.rs as
//     the file.
//   * SerialFifo path is UNCHANGED (H36 byte-equivalence): the worker
//     scheduler is `WorkerScheduler::Fifo`, max_slots=1 invariant,
//     handle.slot_id is ALWAYS SlotId(0) under SerialFifo (per
//     FifoSchedulerAdapter), so the clamp is GUARANTEED inactive.
//   * SlotAware + SlotId(0) for Qwen35 ALSO routes through the
//     existing per-request alloc path (H39 first-slot pin) — the
//     persistent cache `Qwen35LoadedModel::persistent_kv_cache` is
//     `Some(cache)` after spawn but NOT yet consulted (deferred to
//     iter-C2d-cont-kernel).
//
// Skip-mode rationale: per CLAUDE.md "no model load" + "no cargo
// build" constraints, these tests do NOT spawn a real Engine; they are
// either source-grep pins on `worker_run` OR type-level pins on the
// typed-error variants. The full SlotAware-decode end-to-end witness
// requires Path A landing in iter-C2d-cont-kernel + a real GGUF.
// ---------------------------------------------------------------------------
#[cfg(test)]
mod adr040_phase_c_iter2d_cont_qwen35_slot_aware_tests {
    use super::*;

    /// **H36 (skip-mode)** — SerialFifo Qwen35 worker arm remains
    /// byte-equivalent post-C2d-cont. Source-grep pin: the
    /// `Request::Generate` worker arm's Qwen35 dispatch STILL routes
    /// through `generate_qwen35_once` (which calls
    /// `alloc_kv_cache_for_request` internally — the pre-C2d-cont
    /// shape). The clamp added below the Gemma 4 clamp is `handle.
    /// slot_id != SlotId(0)`; under SerialFifo the FifoSchedulerAdapter
    /// always hands out SlotId(0), so the clamp is unreachable in the
    /// SerialFifo arm.
    ///
    /// Mirrors H28's source-grep discipline.
    #[test]
    fn h36_serial_fifo_qwen35_worker_arm_byte_equivalent_post_c2d_cont() {
        let src = include_str!("engine.rs");
        // Find the worker_run function body.
        let body_start = src
            .find("fn worker_run(")
            .expect("H36: worker_run entry not found");
        // Bound the search to the worker_run function body — use the
        // sentinel of the next top-level item.
        let body_after = &src[body_start..];
        let body_end_off = body_after
            .find("\n// The worker thread for `LoadedModel::Qwen35` returns a sentinel error")
            .or_else(|| body_after.find("\n/// Worker-thread entry point"))
            .unwrap_or(body_after.len().min(200_000));
        let body = &body_after[..body_end_off];
        // The Request::Generate arm still calls `generate_qwen35_once`
        // (the pre-C2d-cont production path). Source-grep pin.
        assert!(
            body.contains("super::engine_qwen35::generate_qwen35_once("),
            "H36 FALSIFIED: post-C2d-cont worker_run Qwen35 Request::Generate \
             arm no longer routes through `generate_qwen35_once`. SerialFifo \
             byte-equivalence with pre-C2d-cont is BROKEN. The Path B clamp \
             must NOT replace the existing forward call — it must SIBLING it \
             below the Gemma 4 clamp."
        );
        // The Embed arm still calls `embed_qwen35` (pre-C2d-cont
        // production path).
        assert!(
            body.contains("super::engine_qwen35::embed_qwen35("),
            "H36 sanity: Embed Qwen35 dispatch still routes through \
             `embed_qwen35` (pre-C2d-cont surface). If this fails the \
             SerialFifo embed byte-equivalence is broken."
        );
        // The streaming arm still calls
        // `generate_stream_qwen35_once_extended`.
        assert!(
            body.contains("super::engine_qwen35::generate_stream_qwen35_once_extended("),
            "H36 sanity: GenerateStream Qwen35 dispatch still routes \
             through `generate_stream_qwen35_once_extended` (pre-C2d-cont \
             surface). SerialFifo streaming byte-equivalence is broken."
        );
    }

    /// **H37 (skip-mode pin, REVISED iter-C2d-cont-kernel iter-1 2026-05-29)** —
    /// historical Display round-trip pin preserved for shape stability
    /// (the label string was the C2d-cont §6.1.24 Path B clamp surface);
    /// iter-C2d-cont-kernel iter-1 REPLACES the production Generate-arm
    /// clamp with the actual lift via
    /// `generate_qwen35_once_slot_aware` per §6.1.27. The OTHER 3 worker
    /// arms (GenerateStream / Embed / GenerateWithSoftTokens) still
    /// carry a relabeled clamp with `iter-C2d-cont-kernel-iter-{2,3,4}`
    /// per ADR-040 §6.1.27 cites. See H51 + H52 for the iter-1 lift's
    /// behavioural pins; this test preserves the original Display
    /// round-trip for the label-format contract.
    #[test]
    fn h37_capability_unsupported_label_names_iter_c2d_cont_kernel_for_qwen35() {
        let err = MultiSeqError::CapabilityUnsupported {
            capability:
                "qwen35-forward-gpu-last-logits-slot-N (iter-C2d-cont-kernel per ADR-040 §6.1.24 — gated on persistent_kv_cache worker hot path lift + slot_id threading through Qwen35Model::forward_gpu_last_logits in src/serve/api/engine_qwen35.rs)",
        };
        let msg = format!("{}", err);
        assert!(
            msg.contains("qwen35-forward-gpu-last-logits-slot-N"),
            "H37 FALSIFIED: typed-deferral label must name the deferred \
             capability (qwen35 forward path) for operator-actionable \
             diagnostics. Got: {msg}"
        );
        assert!(
            msg.contains("iter-C2d-cont-kernel"),
            "H37 FALSIFIED: typed-deferral label must name the implementing \
             iter (iter-C2d-cont-kernel) so operator log greps land on \
             the right pin pointer. Got: {msg}"
        );
        assert!(
            msg.contains("persistent_kv_cache"),
            "H37 FALSIFIED: typed-deferral label must name the gating \
             primitive (persistent_kv_cache); without this cite, a \
             future iter that lifts the deferral cannot grep for what \
             unblocks it. Got: {msg}"
        );
        assert!(
            msg.contains("engine_qwen35.rs"),
            "H37 FALSIFIED: typed-deferral label must name the file \
             that needs the worker-hot-path lift — Chesterton's fence \
             on the worker arm's string-prefix contract that handlers \
             string-match against. Got: {msg}"
        );
    }

    /// **H38 (skip-mode, REVISED iter-C2d-cont-kernel iter-1 2026-05-29)** —
    /// post-iter-1 coverage pin: the iter-C2d-cont-kernel-iter-{2,3,4}
    /// relabeled clamps still appear in the 3 worker arms
    /// (GenerateStream / Embed / GenerateWithSoftTokens) that DID NOT
    /// land in iter-1 (Path B for the streaming + embed + soft-token
    /// surfaces per §6.1.27). The Generate arm's clamp at iter-1 is
    /// REPLACED by the actual lift via
    /// `generate_qwen35_once_slot_aware`. Defends the deferral
    /// discipline: each surviving deferral has an iter-N label naming
    /// the implementing iter; the lifted Generate arm has the
    /// structural marker pin via H51 (no clamp at Generate).
    ///
    /// Pin: `rollback_la_to` is still NOT called from `worker_run` —
    /// the spec-decode capture-rollback path is iter-B4d scope per
    /// §6.1.26 deferrals matrix. The iter-1 slot-aware Generate arm
    /// uses `reset_for_slot(slot_id)` (the per-slot reset for the
    /// non-spec-decode generate path), NOT `rollback_la_to` (which
    /// requires `ensure_la_capture` only allocated in spec-decode).
    #[test]
    fn h38_typed_deferral_label_present_in_all_four_worker_arms_and_rollback_la_to_not_yet_called()
    {
        let src = include_str!("engine.rs");
        let body_start = src
            .find("fn worker_run(")
            .expect("H38: worker_run entry not found");
        let body_after = &src[body_start..];
        let body_end_off = body_after
            .find("\n// The worker thread for `LoadedModel::Qwen35` returns a sentinel error")
            .or_else(|| body_after.find("\n/// Worker-thread entry point"))
            .unwrap_or(body_after.len().min(200_000));
        let body = &body_after[..body_end_off];

        // Post-iter-3 (§6.1.29, 2026-05-30): the original
        // `iter-C2d-cont-kernel per ADR-040 §6.1.24` clamp label is
        // REMOVED from production (the Generate / GenerateStream /
        // Embed arms each landed their actual lifts at iter-1 / iter-2
        // / iter-3 respectively). Only the iter-4 (GenerateWithSoftTokens)
        // clamp remains. Count substring `iter-C2d-cont-kernel-iter-`
        // in the worker_run body — should be ≥1 (the surviving iter-4
        // clamp + the iter-2 / iter-4 lift-fork comments naming the
        // sequencing). The historical ≥3 assertion reflected iter-1's
        // state; iter-2 + iter-3 legitimately narrow the surviving
        // surface.
        let kernel_iter_label = "iter-C2d-cont-kernel-iter-";
        let n_kernel_iter = body.matches(kernel_iter_label).count();
        assert!(
            n_kernel_iter >= 1,
            "H38 FALSIFIED: expected at least 1 occurrence of the \
             post-iter-3 `iter-C2d-cont-kernel-iter-` label in worker_run \
             body (the surviving iter-4 GenerateWithSoftTokens clamp). \
             Got {n_kernel_iter}. Drift here means even the surviving \
             iter-4 sub-deferral lost its label."
        );

        // iter-1 lift witness: the worker_run body must contain a
        // call to `generate_qwen35_once_slot_aware` (the new slot-aware
        // Generate-arm routing). Source-grep pin.
        assert!(
            body.contains("generate_qwen35_once_slot_aware"),
            "H38 FALSIFIED: worker_run does NOT call \
             `generate_qwen35_once_slot_aware` — iter-C2d-cont-kernel \
             iter-1 lift did not land in the Generate worker arm. \
             Source-grep against the function name expected since the \
             iter-1 worker-arm site routes through it at SlotId(N>0)."
        );

        // iter-1 also calls `reset_for_slot` via the slot-aware fn —
        // pin the new per-slot reset primitive's presence.
        let any_slot_aware_call = src.contains("generate_qwen35_once_slot_aware(");
        assert!(
            any_slot_aware_call,
            "H38 FALSIFIED: iter-1 slot-aware fn call not found in source"
        );

        // `rollback_la_to` is still structurally ABSENT from worker_run.
        // The iter-1 slot-aware path uses `reset_for_slot` for the
        // non-spec-decode generate path; `rollback_la_to` is reserved
        // for spec-decode capture-state rollback (iter-B4d scope per
        // §6.1.26). When iter-B4d lands the spec-decode slot-aware
        // path, this assertion's predicate must flip to a positive
        // presence pin.
        assert!(
            !body.contains("rollback_la_to"),
            "H38 FALSIFIED: worker_run now calls `rollback_la_to` — \
             this is the iter-B4d spec-decode rollback discipline \
             landing. Update H38 to pin the call shape + remove this \
             structural absence assertion."
        );
    }

    /// **H39 (skip-mode)** — SlotAware + SlotId(0) for Qwen35 routes
    /// through the existing per-request alloc path (NOT the persistent
    /// cache). Source-grep pin that the typed clamp is `handle.slot_id
    /// != SlotId(0)` (NOT `mode is SlotAware`).
    ///
    /// Rationale: under SlotAware with max_slots=N, SlotId(0) is the
    /// first slot handed out by InflightBatchedScheduler. We preserve
    /// byte-equivalence for SlotId(0) at SlotAware by keeping the
    /// existing per-request alloc path — only SlotId(N>0) trips the
    /// Path B clamp. This pin defends against a future drift that
    /// silently extends the clamp to "any SlotAware admission".
    #[test]
    fn h39_qwen35_clamp_is_slot_id_nonzero_only_not_mode_predicate() {
        let src = include_str!("engine.rs");
        let body_start = src
            .find("fn worker_run(")
            .expect("H39: worker_run entry not found");
        let body_after = &src[body_start..];
        let body_end_off = body_after
            .find("\n// The worker thread for `LoadedModel::Qwen35` returns a sentinel error")
            .or_else(|| body_after.find("\n/// Worker-thread entry point"))
            .unwrap_or(body_after.len().min(200_000));
        let body = &body_after[..body_end_off];
        // The Qwen35 clamp predicate is `matches!(loaded,
        // LoadedModel::Qwen35(_)) && handle.slot_id != SlotId(0)`.
        // Source-grep pin: this exact predicate must appear ≥4 times
        // (once per worker arm).
        let predicate = "matches!(loaded, LoadedModel::Qwen35(_)) && handle.slot_id != SlotId(0)";
        let n = body.matches(predicate).count();
        assert!(
            n >= 4,
            "H39 FALSIFIED: expected the Qwen35 clamp predicate \
             `{predicate}` in at least 4 worker arms. Got {n}. \
             Drift here may indicate the clamp extended to all \
             SlotAware admissions (breaking SlotId(0) byte-equivalence) \
             OR was removed from one of the four arms (incomplete \
             coverage)."
        );
    }

    /// **H40 (skip-mode)** — Gemma 4 + Qwen3VL worker arms unchanged
    /// by C2d-cont. Source-grep pin that the Gemma 4 C2c clamp
    /// (`gemma4-forward-prefill-slot-N`) is still present in 4 worker
    /// arms AND that no Qwen3VL clamp was accidentally added (Qwen3VL
    /// SlotAware activation is deferred to a future iter — see
    /// §6.1.22 spawn arm comments).
    #[test]
    fn h40_gemma4_and_qwen3vl_worker_arms_unchanged_by_c2d_cont() {
        let src = include_str!("engine.rs");
        let body_start = src
            .find("fn worker_run(")
            .expect("H40: worker_run entry not found");
        let body_after = &src[body_start..];
        let body_end_off = body_after
            .find("\n// The worker thread for `LoadedModel::Qwen35` returns a sentinel error")
            .or_else(|| body_after.find("\n/// Worker-thread entry point"))
            .unwrap_or(body_after.len().min(200_000));
        let body = &body_after[..body_end_off];

        // Gemma 4 C2c clamp still present in worker arms — the C2c
        // ADR-040 iter-B4c-kernel iter-5 (§6.1.37 — TERMINAL Gemma 4
        // worker-arm lift) REVISES H40: post-iter-5 ALL FOUR Gemma 4
        // worker arms are lifted.  The SOLE SURVIVING SoftTokens-arm
        // C2c clamp label `gemma4-forward-prefill-with-soft-tokens-slot-N
        // (iter-C2c-cont` is LEGITIMATELY REMOVED by iter-5.  Sibling-
        // discipline intent ("C2d-cont must NOT accidentally regress
        // C2c") preserved by pinning the iter-5 lift fn is called from
        // worker_run + iter-1's Gemma 4 Generate lift is still called
        // (defends against C2d-cont accidentally regressing the entire
        // Gemma 4 surface).
        assert!(
            body.contains("generate_gemma4_once_with_soft_tokens_slot_aware("),
            "H40 FALSIFIED (post-iter-5 revision per §6.1.37): \
             Gemma 4 iter-5 TERMINAL SoftTokens lift fn \
             `generate_gemma4_once_with_soft_tokens_slot_aware` is \
             NOT called from worker_run. C2d-cont accidentally \
             regressed the iter-5 §6.1.37 lift."
        );
        assert!(
            body.contains("generate_gemma4_once_slot_aware("),
            "H40 FALSIFIED (post-iter-5 revision per §6.1.37): \
             Gemma 4 iter-1 Generate lift fn \
             `generate_gemma4_once_slot_aware` is NOT called from \
             worker_run. C2d-cont accidentally regressed the iter-1 \
             §6.1.31 lift."
        );

        // ADR-040 iter-C2e (2026-05-30 §6.1.52) — Qwen3VL SlotAware
        // spawn-arm SHIPPED.  The worker_run body now CONTAINS the
        // `matches!(loaded, LoadedModel::Qwen3VlText(_))` clamp at
        // SlotId(N>0) for each of the four arms (Generate /
        // GenerateStream / Embed / GenerateWithSoftTokens) per
        // §6.1.52. Sibling discipline pin: C2d-cont must not REMOVE
        // the C2e Qwen3VL clamp (and must not have added it
        // pre-C2e — the C2d-cont commit `f886f45f` predates C2e).
        // Post-C2e source ordering: the C2e Qwen3VL Generate clamp
        // sits BELOW the C2d Qwen35 Generate clamp in source order,
        // mirroring the spawn_with_mode dispatch order
        // (Gemma → Qwen35 → Qwen3VlText).
        assert!(
            body.contains(
                "matches!(loaded, LoadedModel::Qwen3VlText(_)) && handle.slot_id != SlotId(0)"
            ),
            "H40 FALSIFIED (post-C2e revision per §6.1.52): Qwen3VL \
             clamp is MISSING from worker_run. iter-C2e SHIPPED 2026-05-30 \
             flipping the Qwen3VL SlotAware spawn arm to `Ok(Engine)` AND \
             adding the four worker-arm clamps (one per Request variant). \
             C2d-cont must NOT regress the C2e Qwen3VL clamp."
        );
    }
}

// ---------------------------------------------------------------------------
// ADR-040 Phase C iter-C2d-cont-kernel iter-1 (2026-05-29) — Qwen35 worker
// hot path Generate-arm lift onto the persistent multi-seq HybridKvCache.
//
// This module pins H51-H57 — the iter-1 lift assertions per ADR §6.1.27.
// The C2d-cont Path B clamp (§6.1.24) at the Generate worker arm is
// REPLACED with a real persistent-cache routing call to
// `engine_qwen35::generate_qwen35_once_slot_aware`; the other 3 worker
// arms (GenerateStream / Embed / GenerateWithSoftTokens) retain a
// relabeled clamp with iter-C2d-cont-kernel-iter-{2,3,4} per §6.1.27
// cites — those iters are the typed sub-deferrals iter-1 leaves in
// place.
//
// Tests (all skip-mode per CLAUDE.md "no model load" + "no cargo build"):
//   H51 — SerialFifo / SlotId(0) byte-equivalence preserved: the
//         worker_run Qwen35 dispatch path under SerialFifo or
//         SlotAware+SlotId(0) still routes through
//         `generate_qwen35_once` (unchanged), NOT
//         `generate_qwen35_once_slot_aware`. Pin via the `if matches!`
//         predicate `&& handle.slot_id != SlotId(0)` source-grep —
//         when this is FALSE, the lift fork doesn't fire and the
//         existing per-request alloc path at the `match &mut loaded`
//         block fires verbatim.
//   H52 — SlotId(N>0) lift landed: worker_run contains a real call to
//         `super::engine_qwen35::generate_qwen35_once_slot_aware(`
//         under the Qwen35 Generate arm. Source-grep pin (mirror of
//         H36's existing pre-iter-1 generate_qwen35_once pin).
//   H53 — persistent-cache field-shape pin: the slot-aware fn's call
//         site `take()`s the persistent cache from `Qwen35LoadedModel.
//         persistent_kv_cache` (Option<HybridKvCache>) + restores it
//         on the OK + Err paths. Source-grep pin on both the take and
//         the put-back assignment.
//   H54 — per-slot reset on completion: the slot-aware fn calls
//         `reset_for_slot(slot_id)` at entry + exit so the persistent
//         cache is request-isolated within the slot. Source-grep pin
//         in engine_qwen35.rs on the function body. Note: this
//         REPLACES the C2d-cont H38 deferral marker that said
//         `rollback_la_to` would be called — the non-spec-decode
//         generate path uses `reset_for_slot` (cursor + linear-attn
//         zero) NOT `rollback_la_to` (which requires
//         `ensure_la_capture` only used in spec-decode per
//         `gpu_full_attn.rs:2705` runtime gate).
//   H55 — typed error on missing persistent_kv_cache: when
//         `persistent_kv_cache.is_none()` at SlotId(N>0) for Qwen35
//         (impossible at runtime per C2d spawn-arm invariant, but
//         defense-in-depth), the worker arm returns a typed
//         `anyhow::Error` with `capability_unsupported:` prefix +
//         operator-grep'able `iter-C2d-cont-kernel iter-1` label +
//         `persistent_kv_cache is None` substring. Source-grep on the
//         worker_run body.
//   H56 — Gemma 4 + Qwen3VL worker arms unchanged by iter-1: mirror
//         of C2d-cont H40 — no Gemma / Qwen3VL clamps modified, no
//         Qwen3VL clamp accidentally added.
//   H57 — historical C2d-cont H37/H38 markers updated honestly: the
//         post-iter-1 source-grep on the original C2d-cont label
//         `iter-C2d-cont-kernel per ADR-040 §6.1.24` in the worker_run
//         body shows ZERO production occurrences (only in test
//         modules); the relabeled iter-C2d-cont-kernel-iter-{2,3,4}
//         labels per §6.1.27 take the role for the remaining 3 arms.
//
// LCP / chunked-prefill / spec-decode are EXPLICITLY out of iter-1 scope
// (each is its own iter-N sub-deferral per §6.1.27 — see the deferrals
// matrix in §6.1.27 for the iter-1 → iter-{2,3,4,LCP,G} sequencing).
// ---------------------------------------------------------------------------
#[cfg(test)]
mod adr040_phase_c_iter_c2d_cont_kernel_iter1_qwen35_tests {
    // No `use super::*;` — all tests are skip-mode source-grep against
    // `include_str!` rather than calling any types in the parent module.

    // ── Helper: snip worker_run body the same way C2d-cont tests do ──
    fn worker_run_body(src: &str) -> &str {
        let body_start = src
            .find("fn worker_run(")
            .expect("iter-1: worker_run entry not found");
        let body_after = &src[body_start..];
        let body_end_off = body_after
            .find("\n// The worker thread for `LoadedModel::Qwen35` returns a sentinel error")
            .or_else(|| body_after.find("\n/// Worker-thread entry point"))
            .unwrap_or(body_after.len().min(200_000));
        &body_after[..body_end_off]
    }

    /// **H51 (skip-mode)** — SerialFifo + SlotId(0) AND SlotAware +
    /// SlotId(0) Qwen35 Generate dispatch is byte-equivalent post-iter-1.
    ///
    /// Source-grep pin: the iter-1 lift fork at the Generate arm uses
    /// the predicate `handle.slot_id != SlotId(0)`. SerialFifo always
    /// hands out SlotId(0) (FifoSchedulerAdapter invariant); SlotAware's
    /// first request also gets SlotId(0). In both cases the predicate
    /// is FALSE → the lift block falls through to the existing
    /// `match &mut loaded { LoadedModel::Qwen35(q) => generate_qwen35_once(..) }`
    /// dispatch, byte-equivalent to pre-iter-1 + pre-C2d-cont.
    ///
    /// Defends the H1 / H2 / H23 / H28 / H36 byte-equivalence chain
    /// that A5* + C2a/C2b + C2d-cont preserved. Same logical contract
    /// as H36 (pre-iter-1), now restated under the iter-1 lift fork.
    #[test]
    fn h51_slot_id_0_qwen35_routes_through_generate_qwen35_once_byte_equivalent() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        // The pre-C2d-cont generate_qwen35_once dispatch must still be
        // reachable from the worker arm (the fallback when the lift
        // predicate is FALSE = SlotId(0)).
        assert!(
            body.contains("super::engine_qwen35::generate_qwen35_once("),
            "H51 FALSIFIED: post-iter-1 worker_run Qwen35 Generate \
             dispatch no longer routes through `generate_qwen35_once` \
             for SlotId(0). The iter-1 lift fork must be ADDITIVE \
             (sibling above the `match &mut loaded` dispatch), NOT \
             REPLACE the SerialFifo / SlotId(0) path. SerialFifo + \
             SlotId(0) byte-equivalence (H36 + H1 + H2) is BROKEN."
        );

        // The lift fork predicate is `slot_id != SlotId(0)` — pin via
        // source-grep that the predicate guards the lift call. Drift
        // here may indicate the predicate accidentally extended to
        // SlotId(0) too.
        assert!(
            body.contains(
                "matches!(loaded, LoadedModel::Qwen35(_)) && handle.slot_id != SlotId(0)"
            ),
            "H51 FALSIFIED: the iter-1 lift predicate at the Generate \
             arm is no longer `slot_id != SlotId(0)`. Drift here means \
             the lift may fire at SlotId(0) too, breaking byte-equivalence."
        );
    }

    /// **H52 (skip-mode)** — iter-1 Generate-arm lift landed at
    /// `worker_run`: the slot-aware fn `generate_qwen35_once_slot_aware`
    /// is called from the worker_run body at the Qwen35 Generate arm.
    /// Source-grep pin (mirror of H36's pre-iter-1 `generate_qwen35_once`
    /// pin, now extended to also pin the new slot-aware entry).
    #[test]
    fn h52_iter1_lift_landed_for_qwen35_generate_arm() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        // The iter-1 lift entry point is the new slot-aware fn.
        // Source-grep pin: the worker_run body MUST call
        // `super::engine_qwen35::generate_qwen35_once_slot_aware(`.
        assert!(
            body.contains("super::engine_qwen35::generate_qwen35_once_slot_aware("),
            "H52 FALSIFIED: iter-1 lift fn \
             `generate_qwen35_once_slot_aware` is NOT called from the \
             worker_run body. The Generate-arm SlotId(N>0) routing is \
             missing — iter-1 didn't actually land. Check the if-block \
             at the Qwen35 Generate arm in src/serve/api/engine.rs::worker_run."
        );

        // The slot-aware fn passes `slot_id` (the SlotId from the
        // admit'd handle), NOT a hard-coded SlotId(0). Pin the
        // threading.
        let lift_block_start = body
            .find("super::engine_qwen35::generate_qwen35_once_slot_aware(")
            .expect("H52: lift call site not found");
        let lift_block_end = body[lift_block_start..]
            .find(");")
            .map(|off| lift_block_start + off + 2)
            .unwrap_or(body.len().min(lift_block_start + 1000));
        let lift_block = &body[lift_block_start..lift_block_end];
        assert!(
            lift_block.contains("slot_id"),
            "H52 FALSIFIED: the lift call site does not pass `slot_id` \
             into `generate_qwen35_once_slot_aware`. The iter-1 lift \
             must thread the admit'd SlotHandle's slot_id into the \
             slot-aware fn (B4b §6.1.20 signature). Got block: {lift_block}"
        );
    }

    /// **H53 (skip-mode)** — persistent-cache `take()` + restore pattern
    /// at the lift call site. Pin both the `q.persistent_kv_cache.take()`
    /// extraction AND the `q.persistent_kv_cache = Some(persistent)`
    /// restoration. This pin prevents two regressions:
    /// (a) caller forgets to put the cache back → next request finds
    ///     `persistent_kv_cache.is_none()` and hits the H55 typed-error
    ///     defense-in-depth path;
    /// (b) caller accidentally clones the cache instead of taking it →
    ///     the persistent cache's per-slot state is not actually
    ///     mutated, defeating cross-request isolation.
    #[test]
    fn h53_lift_call_site_takes_and_restores_persistent_kv_cache() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        assert!(
            body.contains("q.persistent_kv_cache.take()"),
            "H53 FALSIFIED: the iter-1 lift call site does not \
             `take()` the persistent cache out of \
             `Qwen35LoadedModel.persistent_kv_cache`. The take is \
             required to resolve the partial-borrow conflict between \
             `&mut q.persistent_kv_cache` and the dense `&mut q` \
             accesses inside `generate_qwen35_once_slot_aware` (q. \
             lcp_registry, q.prompt_cache, etc.)."
        );

        assert!(
            body.contains("q.persistent_kv_cache = Some(persistent)"),
            "H53 FALSIFIED: the iter-1 lift call site does not put \
             the persistent cache back into `q.persistent_kv_cache` \
             after the slot-aware fn returns. The next request to \
             land at SlotId(N>0) would find `persistent_kv_cache. \
             is_none()` and hit the H55 defense-in-depth typed error \
             — defeats the persistent-cache invariant established by \
             C2d (§6.1.22)."
        );
    }

    /// **H54 (skip-mode)** — per-slot reset on completion via
    /// `reset_for_slot(slot_id)` at entry + exit of the slot-aware fn.
    /// Source-grep pin on `engine_qwen35.rs` for the body of
    /// `generate_qwen35_once_slot_aware`. Note: this REPLACES the
    /// C2d-cont H38 deferral marker that said `rollback_la_to` would
    /// be called — the non-spec-decode generate path uses
    /// `reset_for_slot` (per-slot cursor zero + per-slot linear-attn
    /// zero), NOT `rollback_la_to` (which requires `ensure_la_capture`
    /// only allocated in spec-decode per `gpu_full_attn.rs:2705`).
    #[test]
    fn h54_slot_aware_fn_calls_reset_for_slot_at_entry_and_exit() {
        let src = include_str!("../../inference/models/qwen35/kv_cache.rs");
        assert!(
            src.contains("pub fn reset_for_slot("),
            "H54 FALSIFIED: `HybridKvCache::reset_for_slot` is not \
             defined in src/inference/models/qwen35/kv_cache.rs. \
             iter-1 requires this new per-slot reset primitive."
        );

        let engine_q = include_str!("engine_qwen35.rs");
        let fn_marker = "pub fn generate_qwen35_once_slot_aware(";
        let fn_start = engine_q
            .find(fn_marker)
            .expect("H54: generate_qwen35_once_slot_aware not defined");
        // Locate the fn body — bound by next `pub fn` or end-of-file.
        let body_after = &engine_q[fn_start..];
        let body_end_off = body_after[fn_marker.len()..]
            .find("\npub fn ")
            .map(|off| off + fn_marker.len())
            .unwrap_or(body_after.len().min(50_000));
        let fn_body = &body_after[..body_end_off];

        let reset_calls = fn_body.matches("reset_for_slot(slot_id)").count();
        assert!(
            reset_calls >= 2,
            "H54 FALSIFIED: `generate_qwen35_once_slot_aware` must \
             call `kv_cache.reset_for_slot(slot_id)` at LEAST TWICE \
             (once at entry, once at exit) for request isolation \
             within the persistent cache slot. Got {reset_calls} \
             call(s). Drift here means the persistent cache may carry \
             stale bytes across requests on the same slot — corrupts \
             cross-request linear-attn recurrent state."
        );
    }

    /// **H55 (skip-mode)** — defense-in-depth typed error when
    /// `persistent_kv_cache.is_none()` at SlotId(N>0) for Qwen35.
    /// Source-grep pin on the worker_run body. The error message
    /// MUST contain the `capability_unsupported:` prefix (handler
    /// string-match for HTTP 501 mapping) + the `iter-C2d-cont-kernel
    /// iter-1` operator-grep'able label + the `persistent_kv_cache is
    /// None` description.
    #[test]
    fn h55_lift_handles_persistent_kv_cache_none_with_typed_error() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        // Look at the q.persistent_kv_cache.take() match arm — when
        // None, the lift must surface a typed error (not panic, not
        // silently fall through).
        assert!(
            body.contains("persistent_kv_cache is None"),
            "H55 FALSIFIED: the iter-1 lift call site does not \
             surface a typed `capability_unsupported` error when \
             `persistent_kv_cache.is_none()`. The defense-in-depth \
             check is required: per C2d (§6.1.22), persistent_kv_cache \
             is always Some(cache) under SlotAware Qwen35 spawn, but \
             a future iter that breaks that invariant must surface a \
             typed error (NOT panic) at this site."
        );
        assert!(
            body.contains("capability_unsupported:")
                && body.contains("iter-C2d-cont-kernel iter-1"),
            "H55 FALSIFIED: the None-branch typed error does not \
             carry both `capability_unsupported:` (handler 501 \
             string-prefix per ADR-040 C3 wiring at schema.rs:344) \
             AND `iter-C2d-cont-kernel iter-1` (operator-grep'able \
             label cite). Both required for the operator runbook."
        );
    }

    /// **H56 (skip-mode, REVISED iter-B4c-kernel iter-3 2026-05-30)** —
    /// Gemma 4 + Qwen3VL worker arms unchanged by Qwen35 iter-1.
    /// Mirror of C2d-cont H40.  REVISED: post-iter-B4c-kernel-iter-3
    /// (§6.1.35) the `gemma4-forward-prefill-slot-N` label is REMOVED
    /// from worker_run (iter-1 §6.1.31 lifted the Gemma 4 Generate arm,
    /// iter-3 §6.1.35 lifted the Gemma 4 GenerateStream arm).  The
    /// surviving C2c clamps (Embed + GenerateWithSoftTokens) still
    /// carry the `iter-C2c-cont` prefix — pin via the still-present
    /// Embed label.
    #[test]
    fn h56_gemma4_and_qwen3vl_worker_arms_unchanged_by_iter1() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        // ADR-040 iter-B4c-kernel iter-5 (§6.1.37 — TERMINAL Gemma 4
        // worker-arm lift) REVISES H56: post-iter-5 ALL FOUR Gemma 4
        // worker arms are lifted.  The SoftTokens clamp label is
        // LEGITIMATELY REMOVED.  Sibling-discipline intent ("Qwen35
        // iter-1 must NOT touch Gemma 4 arms — Gemma 4 SlotAware
        // kernel lift is iter-B4c-kernel scope") preserved via the
        // positive assertion that the iter-5 lift fn is called from
        // worker_run.
        assert!(
            body.contains("generate_gemma4_once_with_soft_tokens_slot_aware("),
            "H56 FALSIFIED (post-iter-5 revision per §6.1.37): \
             Gemma 4 iter-5 TERMINAL SoftTokens lift fn \
             `generate_gemma4_once_with_soft_tokens_slot_aware` is \
             NOT called from worker_run. Qwen35 iter-1 must NOT \
             regress iter-5's §6.1.37 lift."
        );

        // The B4c-cited Gemma 4 iter-B4c-kernel label survives in
        // worker_run via comment narration even post-iter-5.
        assert!(
            body.contains("iter-B4c-kernel per ADR-040 §6.1.25")
                || body.contains("iter-B4c-kernel iter-5"),
            "H56 FALSIFIED: Gemma 4 B4c label-refinement cite \
             `iter-B4c-kernel per ADR-040 §6.1.25` AND the iter-5 \
             closure cite `iter-B4c-kernel iter-5` BOTH missing from \
             worker_run. iter-1 must NOT regress B4c §6.1.25 nor the \
             iter-5 TERMINAL lift cite."
        );

        // ADR-040 iter-C2e (2026-05-30 §6.1.52) — Qwen3VL clamp
        // SHIPPED post-iter-1 (C2d-cont-kernel iter-1 commit predates
        // C2e). Sibling discipline pin: Qwen35 iter-1 must not REMOVE
        // the C2e Qwen3VL clamp.
        assert!(
            body.contains(
                "matches!(loaded, LoadedModel::Qwen3VlText(_)) && handle.slot_id != SlotId(0)"
            ),
            "H56 FALSIFIED (post-C2e revision per §6.1.52): Qwen3VL \
             clamp missing from worker_run. iter-C2e SHIPPED 2026-05-30 \
             adds the Qwen3VL clamp at the four worker arms; iter-1 must \
             NOT regress the C2e clamp."
        );
    }

    /// **H57 (skip-mode)** — iter-1 sub-deferrals coverage: the
    /// `iter-C2d-cont-kernel-iter-` substring appears in the worker_run
    /// body at LEAST 3 times (one per remaining clamp:
    /// GenerateStream / Embed / GenerateWithSoftTokens). This pins the
    /// iter-1 → iter-{2,3,4} sequencing per §6.1.27 — each remaining
    /// arm carries a typed sub-deferral label naming its iter-N
    /// implementer.
    ///
    /// Also pins the §6.1.27 ADR closure block exists.
    #[test]
    fn h57_iter1_sub_deferrals_named_for_remaining_three_arms() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        let kernel_iter_label = "iter-C2d-cont-kernel-iter-";
        let n = body.matches(kernel_iter_label).count();
        // Post-iter-3 (§6.1.29, 2026-05-30): the Embed clamp's
        // `iter-C2d-cont-kernel-iter-3` substring was REMOVED (the
        // actual lift landed via `embed_qwen35_slot_aware`). Only the
        // iter-4 (SoftTokens) clamp + lift-fork comments naming iter-2
        // and iter-4 remain. Pin ≥ 1 for the surviving iter-4 clamp;
        // historical sequencing pin on the §6.1.27 + §6.1.28 + §6.1.29
        // closure-block enumeration is preserved below.
        assert!(
            n >= 1,
            "H57 FALSIFIED: expected at least 1 occurrence of \
             `iter-C2d-cont-kernel-iter-` in worker_run body (the \
             surviving iter-4 SoftTokens clamp). Got {n}. Drift here \
             means even the surviving iter-4 sub-deferral lost its \
             label — the §6.1.27 / §6.1.28 / §6.1.29 sequencing is \
             broken."
        );

        // iter-4 label must still be specifically named (post-iter-3
        // surviving clamp).
        let iter4_label = "iter-C2d-cont-kernel-iter-4";
        assert!(
            body.contains(iter4_label),
            "H57 FALSIFIED: sub-deferral label `{iter4_label}` not \
             present in worker_run body. The iter-4 (SoftTokens) clamp \
             must name its specific iter-N implementer per §6.1.27 / \
             §6.1.29."
        );

        // §6.1.27 closure block must exist in the ADR.
        let adr = crate::serve::api::engine::adr040_history_doc() /* iter-230 A1: §6.1.x moved to history (aeb6e87c) */;
        assert!(
            adr.contains("### 6.1.27"),
            "H57 FALSIFIED: ADR §6.1.27 closure block missing. \
             iter-1 must land the closure block in lockstep with the \
             production code change (per ADR-040 §3.7 closure-discipline)."
        );
        // §6.1.27 must name `iter-C2d-cont-kernel iter-1` for the
        // implemented scope.
        let block_marker = "### 6.1.27";
        let block_start = adr.find(block_marker).expect("§6.1.27 marker");
        let block_end_off = adr[block_start..]
            .find("\n### ")
            .or_else(|| adr[block_start..].find("\n---\n"))
            .or_else(|| adr[block_start..].find("\n## "))
            .unwrap_or(adr[block_start..].len().min(20_000));
        let block = &adr[block_start..block_start + block_end_off];
        assert!(
            block.contains("iter-C2d-cont-kernel iter-1"),
            "H57 FALSIFIED: §6.1.27 closure block does not name \
             `iter-C2d-cont-kernel iter-1` — operator-grep'able cite \
             for the iter-1 scope landing."
        );
        // §6.1.27's deferrals matrix (historical body, unchanged by
        // iter-2 + iter-3) must still enumerate iter-2/3/4 sub-deferrals
        // so the operator runbook traces the full sequencing chain.
        for iter_label in [
            "iter-C2d-cont-kernel-iter-2",
            "iter-C2d-cont-kernel-iter-3",
            "iter-C2d-cont-kernel-iter-4",
        ] {
            assert!(
                block.contains(iter_label),
                "H57 FALSIFIED: §6.1.27 closure block does not name \
                 sub-deferral `{iter_label}` — the §6.1.27 deferrals \
                 matrix must enumerate every iter-N sub-deferral so \
                 the operator runbook is complete."
            );
        }
    }
}

// ---------------------------------------------------------------------------
// ADR-040 Phase C iter-C2d-cont-kernel iter-2 (2026-05-30) — Qwen35 worker
// hot path **GenerateStream-arm** lift onto the persistent multi-seq
// `HybridKvCache`. Direct mirror of iter-1 (§6.1.27 Generate arm) for the
// streaming surface.
//
// This module pins H58-H63 — the iter-2 lift assertions per ADR §6.1.28.
// The C2d-cont Path B clamp (§6.1.24) at the GenerateStream worker arm is
// REPLACED with a real persistent-cache routing call to
// `engine_qwen35::generate_stream_qwen35_once_extended_slot_aware`; the
// other 2 worker arms (Embed / GenerateWithSoftTokens) retain a
// relabeled clamp with iter-C2d-cont-kernel-iter-{3,4} per §6.1.27 cites
// (still load-bearing per H57; iter-2's lift narrows the surviving
// surface from 3 arms to 2).
//
// Tests (all skip-mode per CLAUDE.md "no model load" + "no cargo build"):
//   H58 — SerialFifo / SlotId(0) GenerateStream byte-equivalence
//         preserved: the worker_run Qwen35 GenerateStream dispatch path
//         under SerialFifo or SlotAware+SlotId(0) still routes through
//         `generate_stream_qwen35_once_extended` (unchanged), NOT
//         `generate_stream_qwen35_once_extended_slot_aware`. Pin via
//         the `if matches!` predicate `&& handle.slot_id != SlotId(0)`
//         source-grep — when this is FALSE, the lift fork doesn't fire
//         and the existing per-request alloc path at the
//         `match &mut loaded` block fires verbatim. Mirror of H51.
//   H59 — iter-2 lift landed at GenerateStream arm: the worker_run body
//         contains a real call to
//         `super::engine_qwen35::generate_stream_qwen35_once_extended_slot_aware(`
//         under the Qwen35 GenerateStream arm. Source-grep pin (mirror
//         of H52's pre-iter-1 `generate_qwen35_once_slot_aware` pin).
//   H60 — persistent-cache `take()` + restore pattern at the iter-2
//         lift call site (mirror of H53). The lift call site `take()`s
//         the persistent cache from `Qwen35LoadedModel.persistent_kv_cache`
//         + restores it via `q.persistent_kv_cache = Some(persistent)`
//         after the streaming fn returns. Defense against the same
//         two-regression failure modes H53 catches (forgotten put-back;
//         clone-instead-of-take).
//   H61 — per-slot reset at entry + exit of the slot-aware streaming
//         fn (mirror of H54). Source-grep pin in engine_qwen35.rs on
//         the body of `generate_stream_qwen35_once_extended_slot_aware`
//         for ≥2 occurrences of `reset_for_slot(slot_id)`.
//   H62 — Gemma 4 + Qwen3VL + Qwen35 Embed + Qwen35 GenerateWithSoftTokens
//         worker arms unchanged by iter-2: the Gemma 4 C2c/B4c clamps
//         + the Qwen35 iter-3 (Embed) + iter-4 (SoftTokens) clamp
//         labels are still present (iter-2 narrows from 3 surviving
//         clamps to 2, but does not REMOVE iter-3 or iter-4).
//   H63 — SSE event ordering preserved: the slot-aware streaming fn
//         emits per-token `Delta` events through the splitter chain
//         followed by a terminal `Done` event. Source-grep pin on
//         `engine_qwen35.rs` for the per-token Delta emission helpers
//         (mirror of generate_stream_qwen35_once_extended's
//         emit_fragment + send! macro shape) AND a single `Done` emit
//         site at the bottom of the fn.
//
// LCP / chunked-prefill / spec-decode / vision streaming are EXPLICITLY
// out of iter-2 scope (iter-2 follows iter-1's deferral discipline —
// vision streaming surfaces typed error citing iter-4; LCP/chunked
// surface no behaviour because they're disabled in slot-aware mode per
// §6.1.27 iter-LCP).
// ---------------------------------------------------------------------------
#[cfg(test)]
mod adr040_phase_c_iter_c2d_cont_kernel_iter2_qwen35_tests {
    // No `use super::*;` — all tests are skip-mode source-grep against
    // `include_str!` rather than calling any types in the parent module.

    // ── Helper: snip worker_run body the same way iter-1 tests do ──
    fn worker_run_body(src: &str) -> &str {
        let body_start = src
            .find("fn worker_run(")
            .expect("iter-2: worker_run entry not found");
        let body_after = &src[body_start..];
        let body_end_off = body_after
            .find("\n// The worker thread for `LoadedModel::Qwen35` returns a sentinel error")
            .or_else(|| body_after.find("\n/// Worker-thread entry point"))
            .unwrap_or(body_after.len().min(200_000));
        &body_after[..body_end_off]
    }

    /// **H58 (skip-mode)** — SerialFifo + SlotId(0) AND SlotAware +
    /// SlotId(0) Qwen35 GenerateStream dispatch is byte-equivalent
    /// post-iter-2.
    ///
    /// Source-grep pin: the iter-2 lift fork at the GenerateStream
    /// arm uses the predicate `handle.slot_id != SlotId(0)`. SerialFifo
    /// always hands out SlotId(0) (FifoSchedulerAdapter invariant);
    /// SlotAware's first request also gets SlotId(0). In both cases
    /// the predicate is FALSE → the lift block falls through to the
    /// existing
    /// `match &mut loaded { LoadedModel::Qwen35(q) =>
    ///     generate_stream_qwen35_once_extended(..) }` dispatch,
    /// byte-equivalent to pre-iter-2 + pre-C2d-cont.
    ///
    /// Defends the H1 / H2 / H23 / H28 / H36 / H51 byte-equivalence
    /// chain that A5* + C2a/C2b + C2d-cont + iter-1 preserved. Direct
    /// mirror of H51 for the streaming arm.
    #[test]
    fn h58_slot_id_0_qwen35_stream_routes_through_extended_byte_equivalent() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        // The pre-iter-2 generate_stream_qwen35_once_extended dispatch
        // must still be reachable from the worker arm (the fallback
        // when the lift predicate is FALSE = SlotId(0)).
        assert!(
            body.contains("super::engine_qwen35::generate_stream_qwen35_once_extended("),
            "H58 FALSIFIED: post-iter-2 worker_run Qwen35 GenerateStream \
             dispatch no longer routes through \
             `generate_stream_qwen35_once_extended` for SlotId(0). The \
             iter-2 lift fork must be ADDITIVE (sibling above the \
             `match &mut loaded` dispatch), NOT REPLACE the SerialFifo \
             / SlotId(0) path. SerialFifo + SlotId(0) byte-equivalence \
             (H1 / H2 / H51 chain) is BROKEN for the streaming arm."
        );

        // The lift fork predicate at the GenerateStream arm must be
        // `matches!(loaded, LoadedModel::Qwen35(_)) && handle.slot_id != SlotId(0)`
        // — the same shape iter-1 used for the Generate arm (H51 mirror).
        // Pin: at least TWO occurrences of the literal predicate in the
        // worker_run body (one in the Generate arm fork, one in the
        // GenerateStream arm fork).
        let predicate_count = body
            .matches("matches!(loaded, LoadedModel::Qwen35(_)) && handle.slot_id != SlotId(0)")
            .count();
        assert!(
            predicate_count >= 2,
            "H58 FALSIFIED: the iter-2 lift fork predicate \
             `matches!(loaded, LoadedModel::Qwen35(_)) && handle.slot_id != SlotId(0)` \
             must appear at least TWICE in worker_run body (one for \
             iter-1 Generate arm, one for iter-2 GenerateStream arm). \
             Got {predicate_count}. Drift here means the lift may fire \
             at SlotId(0) too, breaking byte-equivalence."
        );
    }

    /// **H59 (skip-mode)** — iter-2 GenerateStream-arm lift landed at
    /// `worker_run`: the slot-aware fn
    /// `generate_stream_qwen35_once_extended_slot_aware` is called from
    /// the worker_run body at the Qwen35 GenerateStream arm. Source-grep
    /// pin (mirror of H52's pre-iter-1
    /// `generate_qwen35_once_slot_aware` pin).
    #[test]
    fn h59_iter2_lift_landed_for_qwen35_generate_stream_arm() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        // The iter-2 lift entry point is the new slot-aware streaming
        // fn. Source-grep pin: the worker_run body MUST call
        // `super::engine_qwen35::generate_stream_qwen35_once_extended_slot_aware(`.
        assert!(
            body.contains("super::engine_qwen35::generate_stream_qwen35_once_extended_slot_aware("),
            "H59 FALSIFIED: iter-2 lift fn \
             `generate_stream_qwen35_once_extended_slot_aware` is NOT \
             called from the worker_run body. The GenerateStream-arm \
             SlotId(N>0) routing is missing — iter-2 didn't actually \
             land. Check the if-block at the Qwen35 GenerateStream arm \
             in src/serve/api/engine.rs::worker_run."
        );

        // The slot-aware fn passes `slot_id` (the SlotId from the
        // admit'd handle), NOT a hard-coded SlotId(0). Pin the
        // threading via substring search inside the lift call block.
        let lift_block_start = body
            .find("super::engine_qwen35::generate_stream_qwen35_once_extended_slot_aware(")
            .expect("H59: lift call site not found");
        let lift_block_end = body[lift_block_start..]
            .find(");")
            .map(|off| lift_block_start + off + 2)
            .unwrap_or(body.len().min(lift_block_start + 2000));
        let lift_block = &body[lift_block_start..lift_block_end];
        assert!(
            lift_block.contains("slot_id"),
            "H59 FALSIFIED: the lift call site does not pass `slot_id` \
             into `generate_stream_qwen35_once_extended_slot_aware`. \
             The iter-2 lift must thread the admit'd SlotHandle's \
             slot_id into the slot-aware fn (B4b §6.1.20 signature). \
             Got block: {lift_block}"
        );
    }

    /// **H60 (skip-mode)** — persistent-cache `take()` + restore pattern
    /// at the iter-2 lift call site (mirror of H53). Pin both the
    /// `q.persistent_kv_cache.take()` extraction AND the
    /// `q.persistent_kv_cache = Some(persistent)` restoration. The
    /// take+restore pattern is required for:
    /// (a) two-iter symmetry — iter-1 already established this pattern
    ///     for the Generate arm; iter-2 must use the same shape so the
    ///     persistent-cache invariant holds across BOTH Generate +
    ///     GenerateStream requests at any slot.
    /// (b) defense against the same two regressions H53 catches —
    ///     forgotten put-back → next request finds
    ///     `persistent_kv_cache.is_none()` and hits the H55-class
    ///     typed-error defense-in-depth path; clone-instead-of-take →
    ///     persistent cache's per-slot state is not actually mutated,
    ///     defeating cross-request isolation.
    ///
    /// iter-2's take+restore is ADDITIVE — the worker_run body has BOTH
    /// the iter-1 take+restore (Generate arm) AND the iter-2 take+restore
    /// (GenerateStream arm). Pin via count ≥ 2 for both take and restore.
    #[test]
    fn h60_lift_call_site_takes_and_restores_persistent_kv_cache() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        let take_count = body.matches("q.persistent_kv_cache.take()").count();
        assert!(
            take_count >= 2,
            "H60 FALSIFIED: the iter-2 lift call site does not \
             `take()` the persistent cache out of \
             `Qwen35LoadedModel.persistent_kv_cache`. Expected at \
             least 2 occurrences of `q.persistent_kv_cache.take()` in \
             worker_run body (one each for iter-1 Generate + iter-2 \
             GenerateStream lift forks); got {take_count}. The take is \
             required to resolve the partial-borrow conflict between \
             `&mut q.persistent_kv_cache` and the dense `&mut q` \
             accesses inside `generate_stream_qwen35_once_extended_slot_aware`."
        );

        let restore_count = body
            .matches("q.persistent_kv_cache = Some(persistent)")
            .count();
        assert!(
            restore_count >= 2,
            "H60 FALSIFIED: the iter-2 lift call site does not put \
             the persistent cache back into `q.persistent_kv_cache` \
             after the slot-aware streaming fn returns. Expected at \
             least 2 occurrences of \
             `q.persistent_kv_cache = Some(persistent)` in worker_run \
             body (one each for iter-1 + iter-2 lift forks); got \
             {restore_count}. The next request to land at SlotId(N>0) \
             would find `persistent_kv_cache.is_none()` and hit the \
             defense-in-depth typed error — defeats the \
             persistent-cache invariant established by C2d (§6.1.22) \
             + iter-1 (§6.1.27)."
        );
    }

    /// **H61 (skip-mode)** — per-slot reset on completion via
    /// `reset_for_slot(slot_id)` at entry + exit of the slot-aware
    /// streaming fn. Source-grep pin on `engine_qwen35.rs` for the
    /// body of `generate_stream_qwen35_once_extended_slot_aware`.
    /// Mirror of H54 for the streaming arm.
    ///
    /// The streaming fn has MORE than 2 reset sites because the
    /// cancellation / error paths also call `reset_for_slot` (the
    /// streaming fn can early-return on client disconnect or decode
    /// failure; each early-return path must reset the slot to avoid
    /// leaking stale bytes to the next request). Pin: ≥ 2 occurrences.
    #[test]
    fn h61_slot_aware_stream_fn_calls_reset_for_slot_at_entry_and_exit() {
        // reset_for_slot primitive must still be defined (iter-1 added it).
        let src = include_str!("../../inference/models/qwen35/kv_cache.rs");
        assert!(
            src.contains("pub fn reset_for_slot("),
            "H61 FALSIFIED: `HybridKvCache::reset_for_slot` is not \
             defined in src/inference/models/qwen35/kv_cache.rs. \
             iter-2 inherits this primitive from iter-1; if it's \
             gone, iter-1 was reverted."
        );

        let engine_q = include_str!("engine_qwen35.rs");
        let fn_marker = "pub fn generate_stream_qwen35_once_extended_slot_aware(";
        let fn_start = engine_q
            .find(fn_marker)
            .expect("H61: generate_stream_qwen35_once_extended_slot_aware not defined");
        // Locate the fn body — bound by next `pub fn` or end-of-file.
        let body_after = &engine_q[fn_start..];
        let body_end_off = body_after[fn_marker.len()..]
            .find("\npub fn ")
            .map(|off| off + fn_marker.len())
            .unwrap_or(body_after.len().min(80_000));
        let fn_body = &body_after[..body_end_off];

        let reset_calls = fn_body.matches("reset_for_slot(slot_id)").count();
        assert!(
            reset_calls >= 2,
            "H61 FALSIFIED: \
             `generate_stream_qwen35_once_extended_slot_aware` must \
             call `kv_cache.reset_for_slot(slot_id)` at LEAST TWICE \
             (once at entry, once at exit) for request isolation \
             within the persistent cache slot. Got {reset_calls} \
             call(s). Drift here means the persistent cache may carry \
             stale bytes across streaming requests on the same slot — \
             corrupts cross-request linear-attn recurrent state."
        );
    }

    /// **H62 (skip-mode, REVISED iter-C2d-cont-kernel iter-3 2026-05-30)** —
    /// Gemma 4 + Qwen3VL + Qwen35 remaining-arm (GenerateWithSoftTokens
    /// only post-iter-3) worker arms unchanged by iter-3. Original
    /// iter-2 H62 docstring pinned "iter-2 narrowed from 3 clamps to 2
    /// (Embed + GenerateWithSoftTokens)". iter-3 narrows further: the
    /// Embed clamp is REMOVED by iter-3's lift (§6.1.29), so only the
    /// iter-4 (GenerateWithSoftTokens) clamp remains in the Qwen35
    /// worker_run surface. H62's sibling-discipline intent ("iter-N did
    /// not regress prior iters' lifts; sub-deferral clamps preserved
    /// for un-lifted arms") is preserved by pinning the SURVIVING
    /// iter-4 clamp + the prior iter-1/iter-2/iter-3 lift fns.
    #[test]
    fn h62_other_worker_arms_unchanged_by_iter2() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        // ADR-040 iter-B4c-kernel iter-5 (§6.1.37 — TERMINAL Gemma 4
        // worker-arm lift) REVISES H62: post-iter-5 ALL FOUR Gemma 4
        // worker arms are lifted.  The SoftTokens clamp label is
        // LEGITIMATELY REMOVED.  Sibling-discipline intent preserved
        // via the positive assertion that the iter-5 lift fn is called
        // from worker_run.
        assert!(
            body.contains("generate_gemma4_once_with_soft_tokens_slot_aware("),
            "H62 FALSIFIED (post-iter-5 revision per §6.1.37): \
             Gemma 4 iter-5 TERMINAL SoftTokens lift fn \
             `generate_gemma4_once_with_soft_tokens_slot_aware` is \
             NOT called from worker_run. Qwen35 iter-3 must NOT \
             regress iter-5's §6.1.37 lift."
        );

        // The B4c-cited Gemma 4 iter-B4c-kernel label survives in
        // worker_run via comment narration even post-iter-5.
        assert!(
            body.contains("iter-B4c-kernel per ADR-040 §6.1.25")
                || body.contains("iter-B4c-kernel iter-5"),
            "H62 FALSIFIED: Gemma 4 B4c label-refinement cite \
             `iter-B4c-kernel per ADR-040 §6.1.25` AND the iter-5 \
             closure cite `iter-B4c-kernel iter-5` BOTH missing from \
             worker_run. iter-3 must NOT regress B4c §6.1.25 nor the \
             iter-5 TERMINAL lift cite."
        );

        // ADR-040 iter-C2e (2026-05-30 §6.1.52) — Qwen3VL clamp
        // SHIPPED post-iter-3.  Sibling discipline pin: Qwen35
        // iter-3 must not REMOVE the C2e Qwen3VL clamp.
        assert!(
            body.contains(
                "matches!(loaded, LoadedModel::Qwen3VlText(_)) && handle.slot_id != SlotId(0)"
            ),
            "H62 FALSIFIED (post-C2e revision per §6.1.52): Qwen3VL \
             clamp missing from worker_run.  iter-C2e SHIPPED 2026-05-30 \
             adds the Qwen3VL clamp at the four worker arms; iter-3 must \
             NOT regress the C2e clamp."
        );

        // Post-iter-3: the Embed clamp's `qwen35-forward-embed-last-
        // slot-N` label is REMOVED (the actual lift landed via
        // `embed_qwen35_slot_aware`). H62's prior assertion that the
        // Embed clamp persisted reflected iter-2's state; iter-3
        // legitimately lifts that arm and removes the label.
        //
        // Post-iter-4 (REVISED 2026-05-30 §6.1.30): the
        // GenerateWithSoftTokens clamp's `qwen35-forward-gpu-with-soft-
        // tokens-slot-N (iter-C2d-cont-kernel-iter-4` label is also
        // REMOVED (iter-4 lifted the soft-token arm via
        // `generate_qwen35_once_with_soft_tokens_slot_aware` +
        // `generate_qwen35_once_with_soft_tokens_and_deepstack_slot_aware`).
        // H62's prior assertion that the SoftTokens clamp persisted
        // reflected iter-3's state; iter-4 legitimately lifts that arm
        // and removes the label. The sibling-discipline intent ("iter-N
        // did not regress prior iters' lifts") is preserved by pinning
        // the iter-1/iter-2/iter-3/iter-4 lift fns below.

        // iter-1 Generate-arm lift fn must still be called (iter-3 must
        // not regress iter-1's Generate lift).
        assert!(
            body.contains("super::engine_qwen35::generate_qwen35_once_slot_aware("),
            "H62 FALSIFIED: iter-1 lift fn \
             `generate_qwen35_once_slot_aware` is NOT called from \
             worker_run. iter-3 must NOT regress iter-1's Generate \
             arm lift (§6.1.27)."
        );

        // iter-2 GenerateStream-arm lift fn must still be called
        // (iter-3 must not regress iter-2's GenerateStream lift).
        assert!(
            body.contains("super::engine_qwen35::generate_stream_qwen35_once_extended_slot_aware("),
            "H62 FALSIFIED: iter-2 lift fn \
             `generate_stream_qwen35_once_extended_slot_aware` is NOT \
             called from worker_run. iter-3 must NOT regress iter-2's \
             GenerateStream arm lift (§6.1.28)."
        );

        // iter-4 GenerateWithSoftTokens-arm lift fn must be called
        // (post-iter-4 §6.1.30: the SoftTokens clamp is replaced by the
        // actual lift; H62 REVISED to pin the iter-4 lift fn is wired
        // into worker_run alongside iter-1/2/3).
        assert!(
            body.contains(
                "super::engine_qwen35::generate_qwen35_once_with_soft_tokens_slot_aware("
            ),
            "H62 FALSIFIED: iter-4 lift fn \
             `generate_qwen35_once_with_soft_tokens_slot_aware` is NOT \
             called from worker_run. iter-4 must land the SoftTokens \
             arm lift (§6.1.30)."
        );
    }

    /// **H63 (skip-mode)** — SSE event ordering preserved in the
    /// slot-aware streaming fn: per-token `Delta` events are emitted
    /// through the splitter chain, followed by a single terminal
    /// `Done` event (or `Error` event on failure). Source-grep pin on
    /// `engine_qwen35.rs` for the slot-aware streaming fn's body:
    /// (a) the `send!` macro (the SSE helper that calls
    ///     `events.blocking_send` + error early-return);
    /// (b) at least one `GenerationEvent::Delta { kind: DeltaKind::Content,`
    ///     emission site (per-token content delta);
    /// (c) exactly one `GenerationEvent::Done {` emission site
    ///     (terminal stream marker).
    ///
    /// This pin defends against two regression classes: (a) iter-2
    /// emitting tokens through a different event variant (e.g. a
    /// custom Stream event), and (b) iter-2 forgetting the terminal
    /// Done emit (which would leave the SSE stream open until client
    /// timeout).
    #[test]
    fn h63_slot_aware_stream_fn_preserves_sse_event_ordering() {
        let engine_q = include_str!("engine_qwen35.rs");
        let fn_marker = "pub fn generate_stream_qwen35_once_extended_slot_aware(";
        let fn_start = engine_q
            .find(fn_marker)
            .expect("H63: generate_stream_qwen35_once_extended_slot_aware not defined");
        let body_after = &engine_q[fn_start..];
        let body_end_off = body_after[fn_marker.len()..]
            .find("\npub fn ")
            .map(|off| off + fn_marker.len())
            .unwrap_or(body_after.len().min(80_000));
        let fn_body = &body_after[..body_end_off];

        // (a) `send!` macro defined inside the fn (the SSE emit helper)
        //     — defense against iter-2 calling events.blocking_send
        //     without the cancellation-counter early-return wiring.
        assert!(
            fn_body.contains("macro_rules! send {"),
            "H63 FALSIFIED: \
             `generate_stream_qwen35_once_extended_slot_aware` body \
             does not define the `send!` macro for SSE emission. The \
             macro must wrap every `events.blocking_send(...)` call to \
             early-return on client-disconnect — mirror of \
             `generate_stream_qwen35_once_extended` shape per §6.1.28."
        );

        // (b) Per-token Content delta emission site.
        assert!(
            fn_body.contains("GenerationEvent::Delta {")
                && fn_body.contains("kind: DeltaKind::Content,"),
            "H63 FALSIFIED: \
             `generate_stream_qwen35_once_extended_slot_aware` body \
             does not emit `GenerationEvent::Delta {{ kind: \
             DeltaKind::Content, ... }}` per-token. Drift here means \
             the slot-aware streaming fn emits tokens through a \
             different event variant — breaks SSE consumer parity \
             with the pre-iter-2 stream shape."
        );

        // (c) Terminal Done emission — exactly one site at the bottom
        //     of the fn (the `send!(GenerationEvent::Done { ... })`
        //     call).
        let done_count = fn_body.matches("GenerationEvent::Done {").count();
        assert!(
            done_count == 1,
            "H63 FALSIFIED: \
             `generate_stream_qwen35_once_extended_slot_aware` body \
             emits {done_count} `GenerationEvent::Done` events; \
             expected exactly 1 (the terminal stream marker at the \
             bottom of the fn). Drift here means the slot-aware \
             streaming fn forgot the terminal Done (leaves SSE open \
             until client timeout) or emits multiple Dones (breaks \
             SSE consumer state-machine)."
        );

        // The Done emit must follow the final reset_for_slot at exit —
        // structural ordering pin: the reset-then-Done sequence is the
        // exit discipline. Find the Done index + the second
        // reset_for_slot occurrence (entry was first); the second
        // must precede Done in source order (textual proxy for
        // runtime order at the happy-path exit).
        let done_idx = fn_body
            .find("GenerationEvent::Done {")
            .expect("H63: Done emit located");
        // Count reset occurrences before Done; for the happy path
        // (entry + exit), entry-reset is before Done by construction;
        // exit-reset is also before Done in source by the exit
        // discipline (reset → final stats build → send! Done).
        let resets_before_done = fn_body[..done_idx]
            .matches("reset_for_slot(slot_id)")
            .count();
        assert!(
            resets_before_done >= 2,
            "H63 FALSIFIED: at the source-order position of the terminal \
             `GenerationEvent::Done` emit, only {resets_before_done} \
             `reset_for_slot(slot_id)` calls precede it. Expected ≥ 2 \
             (entry + exit). Drift here means the exit-reset is AFTER \
             the Done emit (or missing) — breaks the iter-2 exit \
             discipline pinned by H61."
        );
    }
}

// ---------------------------------------------------------------------------
// ADR-040 Phase C iter-C2d-cont-kernel iter-3 (2026-05-30) — Qwen35 worker
// hot path **Embed-arm** lift onto the persistent multi-seq `HybridKvCache`.
// Direct mirror of iter-1 (§6.1.27 Generate arm) + iter-2 (§6.1.28
// GenerateStream arm) for the embed surface.
//
// This module pins H64-H69 — the iter-3 lift assertions per ADR §6.1.29.
// The C2d-cont Path B clamp (§6.1.24) at the Embed worker arm is REPLACED
// with a real persistent-cache routing call to
// `engine_qwen35::embed_qwen35_slot_aware`; the remaining 1 worker arm
// (GenerateWithSoftTokens) retains a relabeled clamp with
// `iter-C2d-cont-kernel-iter-4` per §6.1.27 / §6.1.29 cite (still
// load-bearing per H57 / H62; iter-3's lift narrows the surviving surface
// from 2 arms to 1).
//
// Tests (all skip-mode per CLAUDE.md "no model load" + "no cargo build"):
//   H64 — SerialFifo / SlotId(0) Embed byte-equivalence preserved: the
//         worker_run Qwen35 Embed dispatch path under SerialFifo or
//         SlotAware+SlotId(0) still routes through the existing
//         `embed_qwen35` dispatch (unchanged), NOT
//         `embed_qwen35_slot_aware`. Pin via the `if matches!`
//         predicate `&& handle.slot_id != SlotId(0)` source-grep — when
//         this is FALSE, the lift fork doesn't fire and the existing
//         non-slot-aware path at the `match &mut loaded` block fires
//         verbatim. Mirror of H51 + H58.
//   H65 — iter-3 lift landed at Embed arm: the worker_run body contains
//         a real call to `super::engine_qwen35::embed_qwen35_slot_aware(`
//         under the Qwen35 Embed arm. Source-grep pin (mirror of
//         H52 / H59).
//   H66 — persistent-cache `take()` + restore pattern at the iter-3
//         lift call site (mirror of H53 / H60). The lift call site
//         `take()`s the persistent cache from
//         `Qwen35LoadedModel.persistent_kv_cache` + restores it via
//         `q.persistent_kv_cache = Some(persistent)` after the
//         slot-aware embed fn returns. Defense against the same
//         regression failure modes H53 / H60 catch (forgotten put-back;
//         clone-instead-of-take). Pin via count ≥ 3 for both take and
//         restore (iter-1 Generate + iter-2 GenerateStream + iter-3
//         Embed).
//   H67 — per-slot reset at entry + exit of the slot-aware embed fn
//         (mirror of H54 / H61). Source-grep pin in engine_qwen35.rs on
//         the body of `embed_qwen35_slot_aware` for ≥2 occurrences of
//         `reset_for_slot(slot_id)`.
//   H68 — Gemma 4 + Qwen3VL + Qwen35 GenerateWithSoftTokens worker arms
//         unchanged by iter-3: the Gemma 4 C2c/B4c clamps + the Qwen35
//         iter-4 (SoftTokens) clamp label are still present (iter-3
//         narrows from 2 surviving Qwen35 clamps to 1, but does not
//         REMOVE iter-4); iter-1 + iter-2 lift fns must still be called
//         (iter-3 must not regress prior iters' lifts).
//   H69 — embed output vector shape preserved: the slot-aware embed fn
//         calls `forward_embed_last(.., slot_id)` (the B4b §6.1.20
//         signature that returns `Vec<f32>` of length `cfg.hidden_size`
//         after L2 normalization). Source-grep + structural pin: the
//         fn return type is `Result<Vec<f32>>` (NOT `Result<Vec<u32>>`
//         or `Result<GenerationResult>` — distinguishes embed from
//         generate); the fn body calls `forward_embed_last`; the
//         per-slot reset discipline doesn't accidentally truncate the
//         output (the exit-reset runs AFTER the embed call completes).
//
// LCP / chunked-prefill / spec-decode / vision streaming are EXPLICITLY
// out of iter-3 scope (the Embed Request variant does not carry
// `soft_tokens` / `deepstack` / `positions_flat`; there is no embed-
// time vision-augmented input surface today). The embed path has no
// decode loop, so LCP / chunked-prefill don't engage structurally —
// iter-3 is the smallest of the iter-{1,2,3,4} arc.
// ---------------------------------------------------------------------------
#[cfg(test)]
mod adr040_phase_c_iter_c2d_cont_kernel_iter3_qwen35_tests {
    // No `use super::*;` — all tests are skip-mode source-grep against
    // `include_str!` rather than calling any types in the parent module.

    // ── Helper: snip worker_run body the same way iter-1 / iter-2 tests do ──
    fn worker_run_body(src: &str) -> &str {
        let body_start = src
            .find("fn worker_run(")
            .expect("iter-3: worker_run entry not found");
        let body_after = &src[body_start..];
        let body_end_off = body_after
            .find("\n// The worker thread for `LoadedModel::Qwen35` returns a sentinel error")
            .or_else(|| body_after.find("\n/// Worker-thread entry point"))
            .unwrap_or(body_after.len().min(200_000));
        &body_after[..body_end_off]
    }

    /// **H64 (skip-mode)** — SerialFifo + SlotId(0) AND SlotAware +
    /// SlotId(0) Qwen35 Embed dispatch is byte-equivalent post-iter-3.
    ///
    /// Source-grep pin: the iter-3 lift fork at the Embed arm uses the
    /// predicate `handle.slot_id != SlotId(0)`. SerialFifo always hands
    /// out SlotId(0) (FifoSchedulerAdapter invariant); SlotAware's
    /// first request also gets SlotId(0). In both cases the predicate
    /// is FALSE → the lift block falls through to the existing
    /// `match &mut loaded { LoadedModel::Qwen35(q) =>
    ///     embed_qwen35(q, &prompt_tokens) }` dispatch,
    /// byte-equivalent to pre-iter-3 + pre-C2d-cont.
    ///
    /// Defends the H1 / H2 / H23 / H28 / H36 / H51 / H58 byte-
    /// equivalence chain that A5* + C2a/C2b + C2d-cont + iter-1 +
    /// iter-2 preserved. Direct mirror of H51 / H58 for the embed arm.
    #[test]
    fn h64_slot_id_0_qwen35_embed_routes_through_embed_qwen35_byte_equivalent() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        // The pre-iter-3 embed_qwen35 dispatch must still be reachable
        // from the worker arm (the fallback when the lift predicate is
        // FALSE = SlotId(0)).
        assert!(
            body.contains("super::engine_qwen35::embed_qwen35(q, &prompt_tokens)"),
            "H64 FALSIFIED: post-iter-3 worker_run Qwen35 Embed \
             dispatch no longer routes through `embed_qwen35` for \
             SlotId(0). The iter-3 lift fork must be ADDITIVE (sibling \
             above the `match &mut loaded` dispatch), NOT REPLACE the \
             SerialFifo / SlotId(0) path. SerialFifo + SlotId(0) byte-\
             equivalence (H1 / H2 / H51 / H58 chain) is BROKEN for the \
             embed arm."
        );

        // The lift fork predicate at the Embed arm must be
        // `matches!(loaded, LoadedModel::Qwen35(_)) && handle.slot_id != SlotId(0)`
        // — the same shape iter-1 / iter-2 used (H51 / H58 mirror).
        // Pin: at least THREE occurrences of the literal predicate in
        // the worker_run body (one in the Generate arm fork, one in
        // the GenerateStream arm fork, one in the Embed arm fork).
        let predicate_count = body
            .matches("matches!(loaded, LoadedModel::Qwen35(_)) && handle.slot_id != SlotId(0)")
            .count();
        assert!(
            predicate_count >= 3,
            "H64 FALSIFIED: the iter-3 lift fork predicate \
             `matches!(loaded, LoadedModel::Qwen35(_)) && handle.slot_id != SlotId(0)` \
             must appear at least THREE TIMES in worker_run body (one \
             for iter-1 Generate arm, one for iter-2 GenerateStream \
             arm, one for iter-3 Embed arm). Got {predicate_count}. \
             Drift here means the lift may fire at SlotId(0) too, \
             breaking byte-equivalence."
        );
    }

    /// **H65 (skip-mode)** — iter-3 Embed-arm lift landed at
    /// `worker_run`: the slot-aware fn `embed_qwen35_slot_aware` is
    /// called from the worker_run body at the Qwen35 Embed arm.
    /// Source-grep pin (mirror of H52 / H59 lift-witness pin).
    #[test]
    fn h65_iter3_lift_landed_for_qwen35_embed_arm() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        // The iter-3 lift entry point is the new slot-aware embed fn.
        // Source-grep pin: the worker_run body MUST call
        // `super::engine_qwen35::embed_qwen35_slot_aware(`.
        assert!(
            body.contains("super::engine_qwen35::embed_qwen35_slot_aware("),
            "H65 FALSIFIED: iter-3 lift fn `embed_qwen35_slot_aware` \
             is NOT called from the worker_run body. The Embed-arm \
             SlotId(N>0) routing is missing — iter-3 didn't actually \
             land. Check the if-block at the Qwen35 Embed arm in \
             src/serve/api/engine.rs::worker_run."
        );

        // The slot-aware fn passes `slot_id` (the SlotId from the
        // admit'd handle), NOT a hard-coded SlotId(0). Pin the
        // threading via substring search inside the lift call block.
        let lift_block_start = body
            .find("super::engine_qwen35::embed_qwen35_slot_aware(")
            .expect("H65: lift call site not found");
        let lift_block_end = body[lift_block_start..]
            .find(");")
            .map(|off| lift_block_start + off + 2)
            .unwrap_or(body.len().min(lift_block_start + 1000));
        let lift_block = &body[lift_block_start..lift_block_end];
        assert!(
            lift_block.contains("slot_id"),
            "H65 FALSIFIED: the lift call site does not pass `slot_id` \
             into `embed_qwen35_slot_aware`. The iter-3 lift must \
             thread the admit'd SlotHandle's slot_id into the slot-\
             aware fn (B4b §6.1.20 signature). Got block: {lift_block}"
        );
    }

    /// **H66 (skip-mode)** — persistent-cache `take()` + restore pattern
    /// at the iter-3 lift call site (mirror of H53 / H60). Pin both the
    /// `q.persistent_kv_cache.take()` extraction AND the
    /// `q.persistent_kv_cache = Some(persistent)` restoration. The
    /// take+restore pattern is required for:
    /// (a) three-iter symmetry — iter-1 + iter-2 already established
    ///     this pattern for the Generate + GenerateStream arms; iter-3
    ///     must use the same shape so the persistent-cache invariant
    ///     holds across ALL Generate + GenerateStream + Embed requests
    ///     at any slot.
    /// (b) defense against the same two regressions H53 / H60 catch —
    ///     forgotten put-back → next request finds
    ///     `persistent_kv_cache.is_none()` and hits the H55-class
    ///     typed-error defense-in-depth path; clone-instead-of-take →
    ///     persistent cache's per-slot state is not actually mutated,
    ///     defeating cross-request isolation.
    ///
    /// iter-3's take+restore is ADDITIVE — the worker_run body now has
    /// THREE take+restore forks (Generate / GenerateStream / Embed).
    /// Pin via count ≥ 3 for both take and restore.
    #[test]
    fn h66_lift_call_site_takes_and_restores_persistent_kv_cache() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        let take_count = body.matches("q.persistent_kv_cache.take()").count();
        assert!(
            take_count >= 3,
            "H66 FALSIFIED: the iter-3 lift call site does not \
             `take()` the persistent cache out of \
             `Qwen35LoadedModel.persistent_kv_cache`. Expected at \
             least 3 occurrences of `q.persistent_kv_cache.take()` in \
             worker_run body (one each for iter-1 Generate + iter-2 \
             GenerateStream + iter-3 Embed lift forks); got \
             {take_count}. The take is required to resolve the \
             partial-borrow conflict between `&mut q.persistent_kv_cache` \
             and the dense `&mut q` accesses inside \
             `embed_qwen35_slot_aware`."
        );

        let restore_count = body
            .matches("q.persistent_kv_cache = Some(persistent)")
            .count();
        assert!(
            restore_count >= 3,
            "H66 FALSIFIED: the iter-3 lift call site does not put \
             the persistent cache back into `q.persistent_kv_cache` \
             after the slot-aware embed fn returns. Expected at least \
             3 occurrences of `q.persistent_kv_cache = Some(persistent)` \
             in worker_run body (one each for iter-1 + iter-2 + iter-3 \
             lift forks); got {restore_count}. The next request to \
             land at SlotId(N>0) would find `persistent_kv_cache.is_none()` \
             and hit the defense-in-depth typed error — defeats the \
             persistent-cache invariant established by C2d (§6.1.22) + \
             iter-1 (§6.1.27) + iter-2 (§6.1.28)."
        );
    }

    /// **H67 (skip-mode)** — per-slot reset on completion via
    /// `reset_for_slot(slot_id)` at entry + exit of the slot-aware
    /// embed fn. Source-grep pin on `engine_qwen35.rs` for the body of
    /// `embed_qwen35_slot_aware`. Mirror of H54 / H61 for the embed
    /// arm.
    ///
    /// The embed fn has exactly 2 reset sites (entry + exit) — embed
    /// has no decode loop / cancellation paths, so the early-return
    /// reset discipline iter-2 introduced for the streaming fn does
    /// not apply here. Pin: ≥ 2 occurrences.
    #[test]
    fn h67_slot_aware_embed_fn_calls_reset_for_slot_at_entry_and_exit() {
        // reset_for_slot primitive must still be defined (iter-1 added it).
        let src = include_str!("../../inference/models/qwen35/kv_cache.rs");
        assert!(
            src.contains("pub fn reset_for_slot("),
            "H67 FALSIFIED: `HybridKvCache::reset_for_slot` is not \
             defined in src/inference/models/qwen35/kv_cache.rs. \
             iter-3 inherits this primitive from iter-1; if it's \
             gone, iter-1 was reverted."
        );

        let engine_q = include_str!("engine_qwen35.rs");
        let fn_marker = "pub fn embed_qwen35_slot_aware(";
        let fn_start = engine_q
            .find(fn_marker)
            .expect("H67: embed_qwen35_slot_aware not defined");
        // Locate the fn body — bound by next `pub fn` or end-of-file.
        let body_after = &engine_q[fn_start..];
        let body_end_off = body_after[fn_marker.len()..]
            .find("\npub fn ")
            .map(|off| off + fn_marker.len())
            .unwrap_or(body_after.len().min(50_000));
        let fn_body = &body_after[..body_end_off];

        let reset_calls = fn_body.matches("reset_for_slot(slot_id)").count();
        assert!(
            reset_calls >= 2,
            "H67 FALSIFIED: `embed_qwen35_slot_aware` must call \
             `kv_cache.reset_for_slot(slot_id)` at LEAST TWICE (once at \
             entry, once at exit) for request isolation within the \
             persistent cache slot. Got {reset_calls} call(s). Drift \
             here means the persistent cache may carry stale bytes \
             across embed requests on the same slot — corrupts cross-\
             request linear-attn recurrent state."
        );
    }

    /// **H68 (skip-mode)** — Gemma 4 + Qwen3VL + Qwen35
    /// GenerateWithSoftTokens worker arm unchanged by iter-3 + iter-1
    /// + iter-2 lift fns still called. Mirror of H56 / H62 extended for
    /// the iter-3 narrowing: iter-3 replaces the iter-3 Embed clamp,
    /// so only the iter-4 (GenerateWithSoftTokens) clamp remains in
    /// the Qwen35 worker_run surface.
    #[test]
    fn h68_other_worker_arms_unchanged_by_iter3() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        // ADR-040 iter-B4c-kernel iter-5 (§6.1.37 — TERMINAL Gemma 4
        // worker-arm lift) REVISES H68: post-iter-5 ALL FOUR Gemma 4
        // worker arms are lifted.  The SoftTokens clamp label is
        // LEGITIMATELY REMOVED.  Sibling-discipline intent preserved
        // via the positive assertion that the iter-5 lift fn is called
        // from worker_run.
        assert!(
            body.contains("generate_gemma4_once_with_soft_tokens_slot_aware("),
            "H68 FALSIFIED (post-iter-5 revision per §6.1.37): \
             Gemma 4 iter-5 TERMINAL SoftTokens lift fn \
             `generate_gemma4_once_with_soft_tokens_slot_aware` is \
             NOT called from worker_run. Qwen35 iter-3 must NOT \
             regress iter-5's §6.1.37 lift."
        );

        // The B4c-cited Gemma 4 iter-B4c-kernel label survives in
        // worker_run via comment narration even post-iter-5.
        assert!(
            body.contains("iter-B4c-kernel per ADR-040 §6.1.25")
                || body.contains("iter-B4c-kernel iter-5"),
            "H68 FALSIFIED: Gemma 4 B4c label-refinement cite \
             `iter-B4c-kernel per ADR-040 §6.1.25` AND the iter-5 \
             closure cite `iter-B4c-kernel iter-5` BOTH missing from \
             worker_run. iter-3 must NOT regress B4c §6.1.25 nor the \
             iter-5 TERMINAL lift cite."
        );

        // ADR-040 iter-C2e (2026-05-30 §6.1.52) — Qwen3VL clamp
        // SHIPPED post-iter-3. Sibling discipline pin: iter-3 must
        // not REMOVE the C2e Qwen3VL clamp.
        assert!(
            body.contains(
                "matches!(loaded, LoadedModel::Qwen3VlText(_)) && handle.slot_id != SlotId(0)"
            ),
            "H68 FALSIFIED (post-C2e revision per §6.1.52): Qwen3VL \
             clamp missing from worker_run. iter-C2e SHIPPED 2026-05-30 \
             adds the Qwen3VL clamp at the four worker arms; iter-3 must \
             NOT regress the C2e clamp."
        );

        // Post-iter-4 (REVISED 2026-05-30 §6.1.30): the
        // GenerateWithSoftTokens clamp's `qwen35-forward-gpu-with-soft-
        // tokens-slot-N (iter-C2d-cont-kernel-iter-4` label is REMOVED
        // (iter-4 lifted the soft-token arm via
        // `generate_qwen35_once_with_soft_tokens_slot_aware`). H68's
        // prior assertion that the clamp persisted reflected iter-3's
        // state; iter-4 legitimately lifts that arm and removes the
        // label. The sibling-discipline intent ("iter-N did not regress
        // prior iters' lifts") is preserved by pinning iter-1/2/3/4
        // lift fns are all called.

        // iter-1 Generate-arm lift fn must still be called (iter-4
        // must not regress iter-1's Generate lift).
        assert!(
            body.contains("super::engine_qwen35::generate_qwen35_once_slot_aware("),
            "H68 FALSIFIED: iter-1 lift fn \
             `generate_qwen35_once_slot_aware` is NOT called from \
             worker_run. iter-4 must NOT regress iter-1's Generate \
             arm lift (§6.1.27)."
        );

        // iter-2 GenerateStream-arm lift fn must still be called
        // (iter-4 must not regress iter-2's GenerateStream lift).
        assert!(
            body.contains("super::engine_qwen35::generate_stream_qwen35_once_extended_slot_aware("),
            "H68 FALSIFIED: iter-2 lift fn \
             `generate_stream_qwen35_once_extended_slot_aware` is NOT \
             called from worker_run. iter-4 must NOT regress iter-2's \
             GenerateStream arm lift (§6.1.28)."
        );

        // iter-3 Embed-arm lift fn must still be called (iter-4 must
        // not regress iter-3's Embed lift).
        assert!(
            body.contains("super::engine_qwen35::embed_qwen35_slot_aware("),
            "H68 FALSIFIED: iter-3 lift fn `embed_qwen35_slot_aware` \
             is NOT called from worker_run. iter-4 must NOT regress \
             iter-3's Embed arm lift (§6.1.29)."
        );

        // iter-4 GenerateWithSoftTokens-arm lift fn must be called
        // (post-iter-4 §6.1.30: the SoftTokens clamp is replaced by the
        // actual lift; H68 REVISED to pin the iter-4 lift fn is wired
        // into worker_run alongside iter-1/2/3).
        assert!(
            body.contains(
                "super::engine_qwen35::generate_qwen35_once_with_soft_tokens_slot_aware("
            ),
            "H68 FALSIFIED: iter-4 lift fn \
             `generate_qwen35_once_with_soft_tokens_slot_aware` is NOT \
             called from worker_run. iter-4 must land the SoftTokens \
             arm lift (§6.1.30)."
        );
    }

    /// **H69 (skip-mode)** — embedding vector output shape preserved
    /// by the slot-aware embed fn. Source-grep + structural pin on
    /// `engine_qwen35.rs` for the body of `embed_qwen35_slot_aware`:
    /// (a) the fn signature returns `Result<Vec<f32>>` (NOT
    ///     `Result<GenerationResult>` or `Result<Vec<u32>>`);
    /// (b) the fn body calls `forward_embed_last(prompt_tokens,
    ///     &positions, kv_cache, slot_id)` — the B4b §6.1.20 slot-
    ///     aware signature that returns the L2-normalized `cfg.hidden_
    ///     size`-length vector (the byte-equivalence baseline);
    /// (c) the exit-reset call runs AFTER the embed forward call (so
    ///     the embed result is not accidentally truncated by the
    ///     reset; the reset is per-slot KV state, not per-fn output).
    ///
    /// This pin defends against two regression classes: (a) iter-3
    /// returning a `GenerationResult` (decode-shaped surface) which
    /// would break the embed-as-vector contract handlers depend on,
    /// and (b) iter-3 inverting the reset/embed order (resetting
    /// AFTER the forward but discarding the output, or resetting
    /// BEFORE entry and BEFORE forward only — both break the per-slot
    /// isolation invariant).
    #[test]
    fn h69_slot_aware_embed_fn_preserves_embedding_vector_shape() {
        let engine_q = include_str!("engine_qwen35.rs");
        let fn_marker = "pub fn embed_qwen35_slot_aware(";
        let fn_start = engine_q
            .find(fn_marker)
            .expect("H69: embed_qwen35_slot_aware not defined");
        // Locate the fn body — bound by next `pub fn` or end-of-file.
        let body_after = &engine_q[fn_start..];
        let body_end_off = body_after[fn_marker.len()..]
            .find("\npub fn ")
            .map(|off| off + fn_marker.len())
            .unwrap_or(body_after.len().min(50_000));
        let fn_body = &body_after[..body_end_off];

        // (a) Return type is `Result<Vec<f32>>` — distinguishes embed
        // from generate (which returns `Result<GenerationResult>`).
        assert!(
            fn_body.contains("-> Result<Vec<f32>>"),
            "H69 FALSIFIED: `embed_qwen35_slot_aware` return type is \
             not `Result<Vec<f32>>`. The embed surface returns the L2-\
             normalized hidden vector (length `cfg.hidden_size`); a \
             different return type breaks the embed-as-vector contract \
             handlers depend on. Mirror of `embed_qwen35`'s shape."
        );

        // (b) The fn body calls `forward_embed_last` — the B4b §6.1.20
        // slot-aware signature that produces the L2-normalized
        // hidden_size-length vector.
        assert!(
            fn_body.contains("forward_embed_last(prompt_tokens"),
            "H69 FALSIFIED: `embed_qwen35_slot_aware` does not call \
             `forward_embed_last(prompt_tokens, ...)`. The embed \
             surface must route through the B4b §6.1.20 slot-aware \
             forward path; calling a different forward fn would break \
             the embed-as-vector byte-equivalence baseline (the L2 \
             normalization happens INSIDE `forward_embed_last`)."
        );

        // (c) The exit-reset call runs AFTER the embed forward call.
        // Source-order pin: the LAST `reset_for_slot(slot_id)` in the
        // body must appear AFTER `forward_embed_last`. Otherwise the
        // exit-reset is misplaced.
        let last_reset = fn_body
            .rfind("reset_for_slot(slot_id)")
            .expect("H69: at least one reset_for_slot(slot_id) call expected");
        let forward_pos = fn_body
            .find("forward_embed_last(prompt_tokens")
            .expect("H69: forward_embed_last call site expected");
        assert!(
            last_reset > forward_pos,
            "H69 FALSIFIED: the LAST `reset_for_slot(slot_id)` call \
             (source-order position {last_reset}) appears BEFORE the \
             `forward_embed_last(prompt_tokens, ...)` call (source-\
             order position {forward_pos}). The exit-reset MUST run \
             AFTER the embed forward so the per-slot cleanup happens \
             on the way out (mirrors iter-1 / iter-2 exit-reset \
             discipline). Inverting the order breaks per-slot \
             isolation for the next request."
        );
    }
}

// ---------------------------------------------------------------------------
// ADR-040 Phase C iter-C2d-cont-kernel iter-4 (2026-05-30) — Qwen35 worker
// hot path **GenerateWithSoftTokens-arm** + **vision-augmented streaming**
// lift onto the persistent multi-seq `HybridKvCache`. TERMINAL lift in
// the Qwen35 worker-arm arc — direct mirror of iter-1 (§6.1.27 Generate
// arm) + iter-2 (§6.1.28 GenerateStream arm) + iter-3 (§6.1.29 Embed arm)
// for the vision-aware soft-token surface.
//
// This module pins H70-H76 — the iter-4 lift assertions per ADR §6.1.30.
// The C2d-cont Path B clamp (§6.1.24) at the GenerateWithSoftTokens worker
// arm is REPLACED with a real persistent-cache routing call to either
// `engine_qwen35::generate_qwen35_once_with_soft_tokens_slot_aware` (soft-
// tokens-only sub-shape) or
// `engine_qwen35::generate_qwen35_once_with_soft_tokens_and_deepstack_slot_aware`
// (deepstack / 3D-positions sub-shape). The iter-2 streaming fn's
// `has_extension == true` typed-error branch is ALSO replaced with the
// real vision-augmented prefill path via
// `forward_gpu_last_logits_with_soft_tokens_and_deepstack(.., slot_id)`.
//
// Post-iter-4 ALL FOUR Qwen35 worker arms (Generate / GenerateStream /
// Embed / GenerateWithSoftTokens) route through the persistent multi-seq
// cache at SlotId(N>0). The remaining sub-deferrals (iter-LCP +
// iter-G) are orthogonal optimizations, NOT arm lifts. Gemma 4
// (iter-B4c-kernel) + Qwen3VL arms unchanged.
//
// Tests (all skip-mode per CLAUDE.md "no model load" + "no cargo build"):
//   H70 — SerialFifo / SlotId(0) SoftTokens byte-equivalence preserved:
//         the worker_run Qwen35 GenerateWithSoftTokens dispatch path
//         under SerialFifo or SlotAware+SlotId(0) still routes through
//         the existing `generate_qwen35_once_with_soft_tokens` /
//         `generate_qwen35_once_with_soft_tokens_and_deepstack` dispatch
//         (unchanged), NOT the slot-aware siblings. Pin via the `if
//         matches!` predicate `&& handle.slot_id != SlotId(0)` source-
//         grep — when this is FALSE, the lift fork doesn't fire and the
//         existing non-slot-aware path at the `match &mut loaded` block
//         fires verbatim. Mirror of H51 / H58 / H64.
//   H71 — iter-4 lift landed at SoftTokens arm: the worker_run body
//         contains real calls to
//         `super::engine_qwen35::generate_qwen35_once_with_soft_tokens_slot_aware(`
//         AND
//         `super::engine_qwen35::generate_qwen35_once_with_soft_tokens_and_deepstack_slot_aware(`
//         under the Qwen35 GenerateWithSoftTokens arm. Source-grep pin
//         (mirror of H52 / H59 / H65). ALSO pins that the iter-4 clamp
//         label `qwen35-forward-gpu-with-soft-tokens-slot-N (iter-C2d-
//         cont-kernel-iter-4` is REMOVED from worker_run (replaced by
//         the lift).
//   H72 — persistent-cache `take()` + restore pattern at the iter-4
//         lift call site (mirror of H53 / H60 / H66). Pin via count ≥ 4
//         for both `q.persistent_kv_cache.take()` and
//         `q.persistent_kv_cache = Some(persistent)` (iter-1 Generate +
//         iter-2 GenerateStream + iter-3 Embed + iter-4 SoftTokens lift
//         forks).
//   H73 — per-slot reset at entry + exit of BOTH slot-aware soft-token
//         fns (mirror of H54 / H61 / H67). Source-grep pin in
//         engine_qwen35.rs on the bodies of
//         `generate_qwen35_once_with_soft_tokens_slot_aware` AND
//         `generate_qwen35_once_with_soft_tokens_and_deepstack_slot_aware`
//         for ≥ 2 occurrences each of `reset_for_slot(slot_id)`.
//   H74 — vision-augmented streaming SlotId(N>0) path lifted: the
//         iter-2 `has_extension` typed-error branch in
//         `generate_stream_qwen35_once_extended_slot_aware` is REPLACED
//         with a real call to
//         `forward_gpu_last_logits_with_soft_tokens_and_deepstack(..,
//         slot_id)`. Source-grep pins on engine_qwen35.rs: (a) the
//         iter-2 typed-error event "vision-augmented streaming slot-
//         aware port is iter-C2d-cont-kernel-iter-4" is REMOVED from
//         the streaming fn body; (b) the streaming fn body now calls
//         `forward_gpu_last_logits_with_soft_tokens_and_deepstack(`;
//         (c) the streaming fn body has a `t_post` computation for
//         post-prefill decode positioning (mirror of the non-streaming
//         deepstack sibling at engine_qwen35.rs:3537).
//   H75 — Gemma 4 + Qwen3VL worker arms unchanged by iter-4: the
//         Gemma 4 C2c/B4c clamp labels are still present; no Qwen3VL
//         clamp accidentally added. iter-1/2/3 lift fns must still be
//         called (iter-4 must not regress any prior lift).
//   H76 — TERMINAL Qwen35 worker-arm sub-deferral pin: NONE of the
//         literal substrings `iter-C2d-cont-kernel-iter-1` /
//         `iter-C2d-cont-kernel-iter-2` / `iter-C2d-cont-kernel-iter-3`
//         / `iter-C2d-cont-kernel-iter-4` appear as a typed-clamp
//         label predicate in worker_run (i.e. NONE appear inside a
//         `MultiSeqError::CapabilityUnsupported { capability: "..." }`
//         block). Surviving sub-deferrals are iter-LCP + iter-G only
//         (orthogonal optimizations, not arm lifts). Historical
//         comments enumerating the iter-N labels ARE allowed (and
//         expected); the pin is on the absence of a CapabilityUnsupported
//         clamp body wrapping these labels. ADR §6.1.30 closure block
//         must exist and name iter-4 SHIPPED.
//
// LCP / chunked-prefill / spec-decode are EXPLICITLY out of iter-4
// scope (LCP/chunked are disabled in slot-aware mode per §6.1.27
// iter-LCP; spec-decode is iter-B4d per §6.1.26).
// ---------------------------------------------------------------------------
#[cfg(test)]
mod adr040_phase_c_iter_c2d_cont_kernel_iter4_qwen35_tests {
    // No `use super::*;` — all tests are skip-mode source-grep against
    // `include_str!` rather than calling any types in the parent module.

    // ── Helper: snip worker_run body the same way iter-1/2/3 tests do ──
    fn worker_run_body(src: &str) -> &str {
        let body_start = src
            .find("fn worker_run(")
            .expect("iter-4: worker_run entry not found");
        let body_after = &src[body_start..];
        let body_end_off = body_after
            .find("\n// The worker thread for `LoadedModel::Qwen35` returns a sentinel error")
            .or_else(|| body_after.find("\n/// Worker-thread entry point"))
            .unwrap_or(body_after.len().min(200_000));
        &body_after[..body_end_off]
    }

    /// **H70 (skip-mode)** — SerialFifo + SlotId(0) AND SlotAware +
    /// SlotId(0) Qwen35 GenerateWithSoftTokens dispatch is byte-
    /// equivalent post-iter-4.
    ///
    /// Source-grep pin: the iter-4 lift fork at the SoftTokens arm uses
    /// the predicate `handle.slot_id != SlotId(0)`. SerialFifo always
    /// hands out SlotId(0) (FifoSchedulerAdapter invariant); SlotAware's
    /// first request also gets SlotId(0). In both cases the predicate
    /// is FALSE → the lift block falls through to the existing
    /// `match &mut loaded { LoadedModel::Qwen35(q) =>
    ///     generate_qwen35_once_with_soft_tokens{,_and_deepstack}(..) }`
    /// dispatch, byte-equivalent to pre-iter-4 + pre-C2d-cont.
    ///
    /// Defends the H1 / H2 / H23 / H28 / H36 / H51 / H58 / H64 byte-
    /// equivalence chain that A5* + C2a/C2b + C2d-cont + iter-1 + iter-2
    /// + iter-3 preserved. Direct mirror of H51 / H58 / H64 for the
    /// SoftTokens arm.
    #[test]
    fn h70_slot_id_0_qwen35_soft_tokens_routes_through_existing_dispatch_byte_equivalent() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        // The pre-iter-4 generate_qwen35_once_with_soft_tokens dispatch
        // must still be reachable from the worker arm (the fallback
        // when the lift predicate is FALSE = SlotId(0)). Pin via
        // substring presence of the soft-tokens-only entry call.
        assert!(
            body.contains("super::engine_qwen35::generate_qwen35_once_with_soft_tokens("),
            "H70 FALSIFIED: post-iter-4 worker_run Qwen35 \
             GenerateWithSoftTokens dispatch no longer routes through \
             `generate_qwen35_once_with_soft_tokens` for SlotId(0). The \
             iter-4 lift fork must be ADDITIVE (sibling above the \
             `match &mut loaded` dispatch), NOT REPLACE the SerialFifo \
             / SlotId(0) path. SerialFifo + SlotId(0) byte-equivalence \
             (H1 / H2 / H51 / H58 / H64 chain) is BROKEN for the \
             SoftTokens arm."
        );

        // The deepstack-aware non-slot-aware dispatch must also still
        // be reachable for the deepstack sub-shape at SlotId(0).
        assert!(
            body.contains(
                "super::engine_qwen35::generate_qwen35_once_with_soft_tokens_and_deepstack("
            ),
            "H70 FALSIFIED: post-iter-4 worker_run Qwen35 \
             GenerateWithSoftTokens deepstack dispatch no longer routes \
             through `generate_qwen35_once_with_soft_tokens_and_deepstack` \
             for SlotId(0). The iter-4 lift fork must be ADDITIVE for \
             the deepstack sub-shape too."
        );

        // The lift fork predicate at the SoftTokens arm must be
        // `matches!(loaded, LoadedModel::Qwen35(_)) && handle.slot_id != SlotId(0)`
        // — the same shape iter-1 / iter-2 / iter-3 used. Pin: at least
        // FOUR occurrences of the literal predicate in the worker_run
        // body (one in each of Generate / GenerateStream / Embed /
        // SoftTokens arm forks).
        let predicate_count = body
            .matches("matches!(loaded, LoadedModel::Qwen35(_)) && handle.slot_id != SlotId(0)")
            .count();
        assert!(
            predicate_count >= 4,
            "H70 FALSIFIED: the iter-4 lift fork predicate \
             `matches!(loaded, LoadedModel::Qwen35(_)) && handle.slot_id != SlotId(0)` \
             must appear at least FOUR TIMES in worker_run body (one \
             for each of iter-1 Generate, iter-2 GenerateStream, iter-3 \
             Embed, iter-4 SoftTokens). Got {predicate_count}. Drift \
             here means the lift may fire at SlotId(0) too, breaking \
             byte-equivalence."
        );
    }

    /// **H71 (skip-mode)** — iter-4 SoftTokens-arm lift landed at
    /// `worker_run`: BOTH slot-aware fns (soft-tokens-only +
    /// deepstack-aware) are called from the worker_run body at the
    /// Qwen35 GenerateWithSoftTokens arm. Source-grep pin (mirror of
    /// H52 / H59 / H65 lift-witness pin) PLUS pin that the iter-4
    /// clamp label is REMOVED.
    #[test]
    fn h71_iter4_lift_landed_for_qwen35_soft_tokens_arm() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        // (a) The iter-4 soft-tokens-only lift entry point.
        assert!(
            body.contains(
                "super::engine_qwen35::generate_qwen35_once_with_soft_tokens_slot_aware("
            ),
            "H71 FALSIFIED: iter-4 lift fn \
             `generate_qwen35_once_with_soft_tokens_slot_aware` is NOT \
             called from the worker_run body. The SoftTokens-arm \
             SlotId(N>0) routing (soft-tokens-only sub-shape) is \
             missing — iter-4 didn't actually land. Check the if-block \
             at the Qwen35 GenerateWithSoftTokens arm in \
             src/serve/api/engine.rs::worker_run."
        );

        // (b) The iter-4 deepstack-aware lift entry point.
        assert!(
            body.contains(
                "super::engine_qwen35::generate_qwen35_once_with_soft_tokens_and_deepstack_slot_aware("
            ),
            "H71 FALSIFIED: iter-4 lift fn \
             `generate_qwen35_once_with_soft_tokens_and_deepstack_slot_aware` \
             is NOT called from the worker_run body. The SoftTokens-arm \
             SlotId(N>0) routing (deepstack / 3D-positions sub-shape) \
             is missing — iter-4 didn't land the deepstack variant."
        );

        // (c) The iter-4 typed-clamp label is REMOVED. The PRE-iter-4
        // clamp had the literal substring
        // `qwen35-forward-gpu-with-soft-tokens-slot-N (iter-C2d-cont-
        // kernel-iter-4 per ADR-040 §6.1.27`. Iter-4 replaces that
        // clamp with the real lift; the substring must no longer
        // appear in a typed-error capability_unsupported context. Use
        // the conservative pin: the literal clamp label string is
        // ABSENT from the worker_run body.
        assert!(
            !body.contains(
                "qwen35-forward-gpu-with-soft-tokens-slot-N (iter-C2d-cont-kernel-iter-4"
            ),
            "H71 FALSIFIED: the pre-iter-4 SoftTokens clamp label \
             `qwen35-forward-gpu-with-soft-tokens-slot-N (iter-C2d-cont-\
             kernel-iter-4` still appears in worker_run. iter-4 must \
             REPLACE this clamp with the real lift; if the substring \
             remains, the lift was added alongside the clamp instead \
             of replacing it."
        );

        // (d) The lift call site passes `slot_id` (the SlotId from the
        // admit'd handle), NOT a hard-coded SlotId(0). Pin via
        // substring search inside both lift call blocks.
        let lift_soft_start = body
            .find("super::engine_qwen35::generate_qwen35_once_with_soft_tokens_slot_aware(")
            .expect("H71: soft-tokens-only lift call site not found");
        let lift_soft_end = body[lift_soft_start..]
            .find(");")
            .map(|off| lift_soft_start + off + 2)
            .unwrap_or(body.len().min(lift_soft_start + 2000));
        let lift_soft_block = &body[lift_soft_start..lift_soft_end];
        assert!(
            lift_soft_block.contains("slot_id"),
            "H71 FALSIFIED: the soft-tokens-only lift call site does \
             not pass `slot_id` into \
             `generate_qwen35_once_with_soft_tokens_slot_aware`. The \
             iter-4 lift must thread the admit'd SlotHandle's slot_id \
             into the slot-aware fn. Got block: {lift_soft_block}"
        );
    }

    /// **H72 (skip-mode)** — persistent-cache `take()` + restore pattern
    /// at the iter-4 lift call site (mirror of H53 / H60 / H66). Pin
    /// both the `q.persistent_kv_cache.take()` extraction AND the
    /// `q.persistent_kv_cache = Some(persistent)` restoration. The
    /// take+restore pattern is required for:
    /// (a) four-iter symmetry — iter-1 + iter-2 + iter-3 already
    ///     established this pattern; iter-4 must use the same shape so
    ///     the persistent-cache invariant holds across ALL Generate +
    ///     GenerateStream + Embed + SoftTokens requests at any slot.
    /// (b) defense against the same two regressions H53 / H60 / H66
    ///     catch — forgotten put-back; clone-instead-of-take.
    ///
    /// iter-4's take+restore is ADDITIVE — the worker_run body now has
    /// FOUR take+restore forks (one per worker arm).
    /// Pin via count ≥ 4 for both take and restore.
    #[test]
    fn h72_lift_call_site_takes_and_restores_persistent_kv_cache() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        let take_count = body.matches("q.persistent_kv_cache.take()").count();
        assert!(
            take_count >= 4,
            "H72 FALSIFIED: the iter-4 lift call site does not \
             `take()` the persistent cache out of \
             `Qwen35LoadedModel.persistent_kv_cache`. Expected at \
             least 4 occurrences of `q.persistent_kv_cache.take()` in \
             worker_run body (one each for iter-1 Generate + iter-2 \
             GenerateStream + iter-3 Embed + iter-4 SoftTokens lift \
             forks); got {take_count}. The take is required to resolve \
             the partial-borrow conflict between \
             `&mut q.persistent_kv_cache` and the dense `&mut q` \
             accesses inside the slot-aware soft-token fns."
        );

        let restore_count = body
            .matches("q.persistent_kv_cache = Some(persistent)")
            .count();
        assert!(
            restore_count >= 4,
            "H72 FALSIFIED: the iter-4 lift call site does not put \
             the persistent cache back into `q.persistent_kv_cache` \
             after the slot-aware soft-token fn returns. Expected at \
             least 4 occurrences of \
             `q.persistent_kv_cache = Some(persistent)` in worker_run \
             body (one each for iter-1 + iter-2 + iter-3 + iter-4 lift \
             forks); got {restore_count}. The next request to land at \
             SlotId(N>0) would find `persistent_kv_cache.is_none()` \
             and hit the defense-in-depth typed error — defeats the \
             persistent-cache invariant established by C2d (§6.1.22) + \
             iter-1 (§6.1.27) + iter-2 (§6.1.28) + iter-3 (§6.1.29)."
        );
    }

    /// **H73 (skip-mode)** — per-slot reset at entry + exit of BOTH
    /// slot-aware soft-token fns via `reset_for_slot(slot_id)` (mirror
    /// of H54 / H61 / H67). Source-grep pin on `engine_qwen35.rs` for
    /// the bodies of `generate_qwen35_once_with_soft_tokens_slot_aware`
    /// AND `generate_qwen35_once_with_soft_tokens_and_deepstack_slot_aware`.
    #[test]
    fn h73_slot_aware_soft_tokens_fns_call_reset_for_slot_at_entry_and_exit() {
        // reset_for_slot primitive must still be defined (iter-1 added it).
        let src = include_str!("../../inference/models/qwen35/kv_cache.rs");
        assert!(
            src.contains("pub fn reset_for_slot("),
            "H73 FALSIFIED: `HybridKvCache::reset_for_slot` is not \
             defined in src/inference/models/qwen35/kv_cache.rs. \
             iter-4 inherits this primitive from iter-1; if it's \
             gone, iter-1 was reverted."
        );

        let engine_q = include_str!("engine_qwen35.rs");

        // (a) soft-tokens-only fn body has ≥ 2 reset_for_slot calls.
        for fn_marker in [
            "pub fn generate_qwen35_once_with_soft_tokens_slot_aware(",
            "pub fn generate_qwen35_once_with_soft_tokens_and_deepstack_slot_aware(",
        ] {
            let fn_start = engine_q
                .find(fn_marker)
                .unwrap_or_else(|| panic!("H73: {fn_marker} not defined"));
            // Locate the fn body — bound by next `pub fn` or end-of-file.
            let body_after = &engine_q[fn_start..];
            let body_end_off = body_after[fn_marker.len()..]
                .find("\npub fn ")
                .map(|off| off + fn_marker.len())
                .unwrap_or(body_after.len().min(60_000));
            let fn_body = &body_after[..body_end_off];

            let reset_calls = fn_body.matches("reset_for_slot(slot_id)").count();
            assert!(
                reset_calls >= 2,
                "H73 FALSIFIED: `{fn_marker}` must call \
                 `kv_cache.reset_for_slot(slot_id)` at LEAST TWICE \
                 (once at entry, once at exit) for request isolation \
                 within the persistent cache slot. Got {reset_calls} \
                 call(s). Drift here means the persistent cache may \
                 carry stale bytes across soft-token requests on the \
                 same slot — corrupts cross-request linear-attn \
                 recurrent state."
            );
        }
    }

    /// **H74 (skip-mode)** — vision-augmented streaming SlotId(N>0)
    /// path lifted: the iter-2 `has_extension` typed-error branch in
    /// `generate_stream_qwen35_once_extended_slot_aware` is REPLACED
    /// with a real call to the soft-tokens-and-deepstack forward.
    /// Source-grep pins on engine_qwen35.rs:
    /// (a) the iter-2 typed-error event "vision-augmented streaming
    ///     slot-aware port is iter-C2d-cont-kernel-iter-4" is REMOVED
    ///     from the streaming fn body;
    /// (b) the streaming fn body now calls
    ///     `forward_gpu_last_logits_with_soft_tokens_and_deepstack(`;
    /// (c) the streaming fn body has a `t_post` computation for
    ///     post-prefill decode positioning.
    #[test]
    fn h74_vision_augmented_streaming_slot_aware_path_lifted() {
        let engine_q = include_str!("engine_qwen35.rs");
        let fn_marker = "pub fn generate_stream_qwen35_once_extended_slot_aware(";
        let fn_start = engine_q
            .find(fn_marker)
            .expect("H74: generate_stream_qwen35_once_extended_slot_aware not defined");
        let body_after = &engine_q[fn_start..];
        let body_end_off = body_after[fn_marker.len()..]
            .find("\npub fn ")
            .map(|off| off + fn_marker.len())
            .unwrap_or(body_after.len().min(100_000));
        let fn_body = &body_after[..body_end_off];

        // (a) The iter-2 typed-error event for has_extension is REMOVED.
        // The pre-iter-4 fn body emitted a typed
        // `capability_unsupported:` error with the substring
        // "vision-augmented streaming slot-aware port is
        // iter-C2d-cont-kernel-iter-4". After iter-4 lands, this
        // substring must NOT appear inside the fn body (the typed-error
        // emit is REPLACED by the actual lift).
        assert!(
            !fn_body.contains(
                "vision-augmented streaming slot-aware port is \
                 iter-C2d-cont-kernel-iter-4"
            ),
            "H74 FALSIFIED: the iter-2 typed-error event \
             `vision-augmented streaming slot-aware port is \
             iter-C2d-cont-kernel-iter-4` still appears in the body of \
             `generate_stream_qwen35_once_extended_slot_aware`. iter-4 \
             must REPLACE this typed-error event with the actual \
             vision-augmented prefill call; if the substring remains, \
             the lift was added alongside the clamp instead of \
             replacing it."
        );

        // (b) The streaming fn body now calls the soft-tokens-and-
        // deepstack forward (the lifted vision-augmented prefill).
        assert!(
            fn_body.contains("forward_gpu_last_logits_with_soft_tokens_and_deepstack("),
            "H74 FALSIFIED: \
             `generate_stream_qwen35_once_extended_slot_aware` body \
             does not call \
             `forward_gpu_last_logits_with_soft_tokens_and_deepstack(`. \
             The iter-4 vision-augmented streaming lift must route \
             `has_extension == true` through this forward (mirror of \
             non-slot-aware sibling at engine_qwen35.rs:4061)."
        );

        // (c) The streaming fn body has a `t_post` computation for
        // post-prefill decode positioning. Mirror of the non-slot-aware
        // sibling at engine_qwen35.rs:4270. The variable name `t_post`
        // is load-bearing — it carries the global temporal counter
        // advance for the vision-augmented path.
        assert!(
            fn_body.contains("let t_post: i32"),
            "H74 FALSIFIED: \
             `generate_stream_qwen35_once_extended_slot_aware` body \
             does not declare a `t_post: i32` local. The iter-4 \
             vision-augmented streaming lift must compute the post-\
             prefill global temporal counter (= `max(positions_flat \
             axis 0) + 1` when supplied; else `prompt_len as i32`) and \
             use it as the decode-step position base. Without t_post, \
             vision-augmented decode steps would use the text-only \
             `prompt_len + step - 1` advance — wrong for image-tail \
             prompts where global temporal != prompt_len."
        );
    }

    /// **H75 (skip-mode)** — Gemma 4 + Qwen3VL worker arms unchanged
    /// by iter-4. Direct mirror of H56 / H62 / H68 extended for the
    /// iter-4 lift. Also pins that iter-1/2/3 lift fns are still
    /// called (iter-4 must not regress any prior lift).
    #[test]
    fn h75_gemma4_and_qwen3vl_worker_arms_unchanged_by_iter4() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        // ADR-040 iter-B4c-kernel iter-5 (§6.1.37 — TERMINAL Gemma 4
        // worker-arm lift) REVISES H75: the SoftTokens-arm clamp label
        // `gemma4-forward-prefill-with-soft-tokens-slot-N (iter-C2c-cont`
        // is LEGITIMATELY REMOVED by iter-5 (it lifted the SoftTokens
        // arm).  Sibling-discipline intent preserved via the positive
        // assertion that the iter-5 lift fn is called from worker_run
        // (mirror of iter-4 §6.1.36's H108 revision pattern that did
        // the same swap when iter-4 lifted the Embed arm).
        assert!(
            body.contains("generate_gemma4_once_with_soft_tokens_slot_aware("),
            "H75 FALSIFIED (post-iter-5 revision per §6.1.37): \
             Gemma 4 iter-5 SoftTokens lift fn \
             `generate_gemma4_once_with_soft_tokens_slot_aware` is \
             NOT called from worker_run. Qwen35 iter-4 must NOT \
             regress iter-5's Gemma 4 SoftTokens-arm lift (§6.1.37)."
        );

        // The B4c-cited Gemma 4 iter-B4c-kernel label survives in
        // worker_run via comment narration even post-iter-5 (the
        // §6.1.25 label-refinement cite is preserved in surviving
        // commentary blocks).
        assert!(
            body.contains("iter-B4c-kernel per ADR-040 §6.1.25")
                || body.contains("iter-B4c-kernel iter-5"),
            "H75 FALSIFIED: Gemma 4 B4c label-refinement cite \
             `iter-B4c-kernel per ADR-040 §6.1.25` AND the iter-5 \
             closure cite `iter-B4c-kernel iter-5` BOTH missing from \
             worker_run. iter-4 must NOT regress B4c §6.1.25 nor the \
             iter-5 TERMINAL lift cite."
        );

        // ADR-040 iter-C2e (2026-05-30 §6.1.52) — Qwen3VL clamp
        // SHIPPED post-iter-4. Sibling discipline pin: iter-4 must
        // not REMOVE the C2e Qwen3VL clamp.
        assert!(
            body.contains(
                "matches!(loaded, LoadedModel::Qwen3VlText(_)) && handle.slot_id != SlotId(0)"
            ),
            "H75 FALSIFIED (post-C2e revision per §6.1.52): Qwen3VL \
             clamp missing from worker_run. iter-C2e SHIPPED 2026-05-30 \
             adds the Qwen3VL clamp at the four worker arms; iter-4 must \
             NOT regress the C2e clamp."
        );

        // iter-1 Generate-arm lift fn must still be called.
        assert!(
            body.contains("super::engine_qwen35::generate_qwen35_once_slot_aware("),
            "H75 FALSIFIED: iter-1 lift fn \
             `generate_qwen35_once_slot_aware` is NOT called from \
             worker_run. iter-4 must NOT regress iter-1's Generate \
             arm lift (§6.1.27)."
        );

        // iter-2 GenerateStream-arm lift fn must still be called.
        assert!(
            body.contains("super::engine_qwen35::generate_stream_qwen35_once_extended_slot_aware("),
            "H75 FALSIFIED: iter-2 lift fn \
             `generate_stream_qwen35_once_extended_slot_aware` is NOT \
             called from worker_run. iter-4 must NOT regress iter-2's \
             GenerateStream arm lift (§6.1.28)."
        );

        // iter-3 Embed-arm lift fn must still be called.
        assert!(
            body.contains("super::engine_qwen35::embed_qwen35_slot_aware("),
            "H75 FALSIFIED: iter-3 lift fn `embed_qwen35_slot_aware` \
             is NOT called from worker_run. iter-4 must NOT regress \
             iter-3's Embed arm lift (§6.1.29)."
        );
    }

    /// **H76 (skip-mode)** — TERMINAL Qwen35 worker-arm sub-deferral
    /// pin: NONE of the literal substrings
    /// `iter-C2d-cont-kernel-iter-1` / `iter-C2d-cont-kernel-iter-2` /
    /// `iter-C2d-cont-kernel-iter-3` / `iter-C2d-cont-kernel-iter-4`
    /// appear as a typed-clamp label predicate in worker_run (i.e. NONE
    /// appear inside a `MultiSeqError::CapabilityUnsupported { capability:
    /// "..." }` block). Surviving sub-deferrals are iter-LCP + iter-G
    /// only (orthogonal optimizations, not arm lifts).
    ///
    /// Historical comments enumerating iter-N labels are allowed (and
    /// expected per the iter-1 §6.1.27 sequencing record); the pin is
    /// on the absence of a CapabilityUnsupported clamp body wrapping
    /// these iter-N labels.
    ///
    /// ALSO pins that ADR §6.1.30 closure block exists + names
    /// `iter-C2d-cont-kernel iter-4` as the SHIPPED scope.
    #[test]
    fn h76_terminal_qwen35_worker_arm_sub_deferrals_pin() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        // (a) Walk the worker_run body looking for
        // `MultiSeqError::CapabilityUnsupported { capability:` blocks
        // wrapping any of the four arm-lift iter-N labels. If any
        // remain, the lift didn't actually land at the worker arm.
        //
        // Note: the Gemma 4 + B4c clamps wrap `iter-C2c-cont` and
        // `iter-B4c-kernel` labels, which are LEGITIMATE surviving
        // sub-deferrals (Gemma 4 arm lifts are gated on B4c-kernel per
        // §6.1.25). The pin is specifically on the Qwen35 iter-N
        // labels (iter-C2d-cont-kernel-iter-1/2/3/4).
        for iter_label in [
            "iter-C2d-cont-kernel-iter-1",
            "iter-C2d-cont-kernel-iter-2",
            "iter-C2d-cont-kernel-iter-3",
            "iter-C2d-cont-kernel-iter-4",
        ] {
            // For each iter-N label, walk every occurrence in
            // worker_run body and verify NONE of them lies within a
            // CapabilityUnsupported clamp block (i.e. between
            // `capability:` and the closing `,` of the same block).
            // The conservative pin: if the substring appears INSIDE a
            // quoted string literal that is the `capability:` value of
            // a `MultiSeqError::CapabilityUnsupported { ... }` block,
            // it's a clamp; comments are fine. The simplest reliable
            // proxy: scan for the *clamp-shaped* surrounding text —
            // `capability:\n... "..iter-N..."`. Per H51-H68 pattern:
            // the clamp string contains `qwen35-forward-...` /
            // `qwen35-stream-...` / `qwen35-embed-...` /
            // `qwen35-forward-gpu-with-soft-tokens-...` PREFIX before
            // the iter-N cite. Pin: for each iter-N label, the prefix
            // family `"qwen35-` followed by anything followed by
            // `(iter-C2d-cont-kernel-iter-N` must NOT appear in the
            // body. This is the exact pre-iter-{1,2,3,4} clamp shape.
            let clamp_shape = format!(
                "(iter-C2d-cont-kernel-iter-{}",
                iter_label
                    .trim_end_matches(|c: char| c.is_ascii_digit() || c == '-')
                    .len()
                    .to_string()
            );
            // Simpler & more reliable: the four pre-iter clamp shapes
            // all had a `qwen35-...-slot-N (iter-C2d-cont-kernel-iter-N`
            // structure. Pin the conservative "no `qwen35-...-slot-N`
            // string immediately followed by `(iter-C2d-cont-kernel-
            // iter-N`" pattern by checking the worker_run body
            // explicitly.
            let _ = clamp_shape;
            let pre_iter_clamp_substr = format!("-slot-N ({iter_label}");
            assert!(
                !body.contains(&pre_iter_clamp_substr),
                "H76 FALSIFIED: the worker_run body still contains the \
                 pre-iter-{n} typed-clamp pattern `-slot-N \
                 ({iter_label}`. Post-iter-4 ALL FOUR Qwen35 worker \
                 arms must route through the persistent multi-seq \
                 cache at SlotId(N>0); no Qwen35 worker arm should \
                 surface a `MultiSeqError::CapabilityUnsupported` \
                 clamp citing these iter-N labels.",
                n = iter_label.chars().last().unwrap_or('?'),
            );
        }

        // (b) Sanity: the lift fns for all four arms are wired into
        // worker_run (defensive — also covered by H75, but H76 makes
        // the terminal-coverage pin self-contained).
        for lift_fn in [
            "super::engine_qwen35::generate_qwen35_once_slot_aware(",
            "super::engine_qwen35::generate_stream_qwen35_once_extended_slot_aware(",
            "super::engine_qwen35::embed_qwen35_slot_aware(",
            "super::engine_qwen35::generate_qwen35_once_with_soft_tokens_slot_aware(",
        ] {
            assert!(
                body.contains(lift_fn),
                "H76 FALSIFIED: the iter-1/2/3/4 lift fn `{lift_fn}` \
                 is NOT called from worker_run. The terminal pin \
                 requires all four arm lifts wired."
            );
        }

        // (c) ADR §6.1.30 closure block must exist + name iter-4.
        let adr = crate::serve::api::engine::adr040_history_doc() /* iter-230 A1: §6.1.x moved to history (aeb6e87c) */;
        assert!(
            adr.contains("### 6.1.30"),
            "H76 FALSIFIED: ADR §6.1.30 closure block missing. iter-4 \
             must land the closure block in lockstep with the \
             production code change (per ADR-040 §3.7 closure-\
             discipline)."
        );
        let block_marker = "### 6.1.30";
        let block_start = adr.find(block_marker).expect("§6.1.30 marker");
        let block_end_off = adr[block_start..]
            .find("\n### ")
            .or_else(|| adr[block_start..].find("\n---\n"))
            .or_else(|| adr[block_start..].find("\n## "))
            .unwrap_or(adr[block_start..].len().min(40_000));
        let block = &adr[block_start..block_start + block_end_off];
        assert!(
            block.contains("iter-C2d-cont-kernel iter-4"),
            "H76 FALSIFIED: §6.1.30 closure block does not name \
             `iter-C2d-cont-kernel iter-4` — operator-grep'able cite \
             for the iter-4 scope landing."
        );
        // The §6.1.30 block must mark this as the TERMINAL Qwen35
        // worker-arm lift (the load-bearing closure-scope pin).
        assert!(
            block.to_ascii_lowercase().contains("terminal"),
            "H76 FALSIFIED: §6.1.30 closure block does not mark iter-4 \
             as the TERMINAL Qwen35 worker-arm lift. The closure must \
             record that post-iter-4 ALL FOUR Qwen35 worker arms route \
             through the persistent multi-seq cache at SlotId(N>0)."
        );
    }
}

// ---------------------------------------------------------------------------
// ADR-040 Phase B iter-4c (B4c) — Gemma 4 worker-arm typed-deferral label
// refinement (Path B symmetric with C2d-cont §6.1.24 for Gemma 4).
//
// Brief: C2c (§6.1.21) shipped the Gemma 4 SlotAware engine spawn arm
// (`Ok(Engine)` with per-layer `MultiSeqHbKvBuffers` provisioning) and
// the four `worker_run` clamps that surface
// `MultiSeqError::CapabilityUnsupported` at `SlotHandle.slot_id !=
// SlotId(0)` for the Gemma 4 architecture. Each clamp's label named
// `iter-C2c-cont per ADR-040 §6.1.21 — gated on B4c kernel slot-offset
// routing through src/serve/forward_prefill.rs`. C2d-cont (§6.1.24)
// then added the symmetric Qwen35 sibling clamps with the now-canonical
// `iter-<phase>-kernel per ADR-040 §<section>` label discipline
// (`iter-C2d-cont-kernel per ADR-040 §6.1.24`).
//
// B4c's job is to **bring the Gemma 4 clamp labels into label-format
// parity with the C2d-cont Qwen35 clamps** so operator log greps + the
// future iter-B4c-kernel implementer find a consistent `iter-<phase>-
// kernel per ADR-040 §<section>` cite across architectures. Path B
// (label refinement only, no kernel work) is mandated by the same
// risk-symmetry reasoning that C2d-cont used: full Gemma 4 `forward_
// prefill.rs` slot threading is ~30 layers × 3 KV variants × `xlen`
// optional ≈ multi-iter work that exceeds the B4c iter ceiling, and
// invalidates the H1/H2 byte-equivalence pin contract until a
// follow-up iter (iter-B4c-kernel) ships the kernel-level routing.
//
// Path decision (this iter): **Path B label refinement**.
//
// Why NOT Path A (full forward_prefill.rs slot lift):
//   1. Surface area: `forward_prefill.rs` + `forward_prefill_batched.rs`
//      thread KV writes through 30 Gemma 4 layers × 3 KV variants
//      (`MultiSeqHbKvBuffers` post-A3a, `HybridKvBuffers` post-A3b
//      iter-1, `DenseKvBuffers` / `MlxKvCache` typed-clamped per
//      A3b iter-1) × the optional `xlen` BF16 buffers. The mechanical
//      refactor footprint exceeds 600 LOC across `forward_prefill.rs` +
//      `forward_prefill_batched.rs` + `gemma4/model.rs` per the
//      §6.1.21 closure block's path-A risk note.
//   2. KV-cache invariants: the existing inline alloc sites at
//      `forward_prefill.rs:843-882`, `forward_prefill_batched.rs:443-
//      475`, and `forward_gpu.rs:443-459` build legacy 3-D
//      `HybridKvBuffers` at implicit `n_seqs=1` (per §6.1.19 A3b iter-1
//      closure). A3a's `alloc_hb_kv_for_layer(.., n_seqs=max_slots)`
//      replacement is gated on Phase B4c per the §6.1.18 closure block.
//      Routing the worker hot path through `Some(persistent_multi_seq)`
//      without the alloc-site refactor would break the byte-equivalence
//      contract for SerialFifo + SlotId(0).
//   3. Byte-equivalence regression risk: H41 (SerialFifo unchanged) +
//      the C2c H23 / C2d-cont H40 pins defend verbatim
//      byte-equivalence with pre-C2c behaviour. Path A invalidates
//      these pins because the kernel slot-offset routing changes the
//      KV-write address calculation even for SlotId(0). The H1/H2
//      byte-equivalence pin arc (A5* + C2a + C2b) explicitly defends
//      against this regression class.
//
// Path B ships the *label refinement* (additive `iter-B4c-kernel per
// ADR-040 §6.1.25` cite appended to the existing `iter-C2c-cont per
// ADR-040 §6.1.21` prefix) — preserving the C2c surface verbatim while
// giving the future iter-B4c-kernel implementer a grep-able pin
// pointer in the typed deferral string. Same dispatch fork shape; same
// 4 worker arms; same `slot_id != SlotId(0)` predicate. The kernel
// work itself is staged as **iter-B4c-kernel** (typed deferral, pinned
// by H42 + H43 label strings — exact mirror of C2d-cont's
// iter-C2d-cont-kernel discipline).
//
// Tests (H41-H45 mirror H36-H40 from C2d-cont 1:1):
//   H41 (skip-mode): SerialFifo Gemma 4 worker arm byte-equivalent
//                    post-B4c. Source-grep pin — Gemma 4 Generate /
//                    GenerateStream / Embed / GenerateWithSoftTokens
//                    arms STILL route through `generate_once` /
//                    `generate_stream_once` / `forward_embed_last` /
//                    `generate_once_with_soft_tokens` (the pre-C2c
//                    production paths). The B4c label refinement is
//                    INSIDE the typed-error string; the dispatch fork
//                    shape is unchanged.
//
//   H42 (skip-mode): typed `MultiSeqError::CapabilityUnsupported`
//                    Display round-trip carries iter-B4c-kernel +
//                    forward_prefill.rs + MultiSeqHbKvBuffers cite
//                    AND preserves the existing iter-C2c-cont + B4c
//                    substrings (C2c surface preservation pin).
//
//   H43 (skip-mode, typed deferral label): the iter-B4c-kernel label
//                                          appears in ≥4 worker arms
//                                          (one per Generate /
//                                          GenerateStream / Embed /
//                                          GenerateWithSoftTokens).
//                                          PLUS: forward_prefill.rs
//                                          slot threading is
//                                          STRUCTURALLY ABSENT from
//                                          `worker_run` today —
//                                          deferral marker for
//                                          iter-B4c-kernel.
//
//   H44 (skip-mode): clamp predicate is `matches!(loaded,
//                    LoadedModel::Gemma(_)) && handle.slot_id !=
//                    SlotId(0)` literal (NOT mode-conditioned);
//                    ≥4 occurrences confirmed. Preserves SerialFifo +
//                    SlotId(0) AND SlotAware + SlotId(0)
//                    byte-equivalence.
//
//   H45 (skip-mode): Qwen35 + Qwen3VL worker arms UNCHANGED by B4c.
//                    C2d-cont's Qwen35 clamps (`qwen35-forward-gpu-
//                    last-logits-slot-N` + `iter-C2d-cont-kernel`
//                    cites) STILL present in 4 worker arms. No Qwen3VL
//                    clamp added (C2e deferral preserved). Mirrors
//                    C2d-cont H40's sibling-discipline pin in reverse.
//
// Path B clamp scope (delta from C2c Gemma 4 pattern):
//   * Each of the 4 worker arms (Generate / GenerateStream / Embed /
//     GenerateWithSoftTokens) now contains the *same* clamp predicate
//     with an *extended* typed-deferral label: the existing
//     `iter-C2c-cont per ADR-040 §6.1.21` cite is preserved as a
//     prefix (so H25 / C2d-cont H40 string-match pins keep passing)
//     followed by ` / iter-B4c-kernel per ADR-040 §6.1.25 — ...`.
//   * SerialFifo path is UNCHANGED (H41 byte-equivalence pin): the
//     scheduler is `WorkerScheduler::Fifo`, max_slots=1, handle.
//     slot_id is ALWAYS SlotId(0), so the clamp is GUARANTEED
//     inactive — same as pre-B4c.
//   * SlotAware + SlotId(0) for Gemma 4 ALSO routes through the
//     existing forward path (H44 first-slot pin) — the persistent
//     `MultiSeqHbKvBuffers` provisioned by C2c (§6.1.21) is `Some`
//     after spawn but the worker hot path still consults it only at
//     the spawn-witness level; iter-B4c-kernel ships the kernel-side
//     routing.
//
// Skip-mode rationale: per CLAUDE.md "no model load" + "no cargo
// build" constraints, these tests do NOT spawn a real Engine; they
// are source-grep pins on `worker_run` + Display round-trip pins on
// the typed-error variants. The full SlotAware-prefill end-to-end
// witness for Gemma 4 requires iter-B4c-kernel landing + a real
// Gemma 4 GGUF (31B production weights are OOM-class on local
// hardware per the C2c `c2c_skip_unless_gated` discipline).
// ---------------------------------------------------------------------------
#[cfg(test)]
mod adr040_phase_b_iter4c_gemma4_slot_aware_tests {
    use super::*;

    /// **H41 (skip-mode)** — SerialFifo Gemma 4 worker arm remains
    /// byte-equivalent post-B4c. Source-grep pin: the
    /// `Request::Generate` worker arm's Gemma 4 dispatch STILL routes
    /// through `generate_once` (which calls the legacy `forward_prefill`
    /// chain internally — the pre-C2c production path). The clamp
    /// predicate is `handle.slot_id != SlotId(0)`; under SerialFifo the
    /// FifoSchedulerAdapter always hands out SlotId(0), so the clamp
    /// is unreachable in the SerialFifo arm regardless of the B4c
    /// label refinement.
    ///
    /// Mirrors C2d-cont H36's source-grep discipline for Gemma 4.
    #[test]
    fn h41_serial_fifo_gemma4_worker_arm_byte_equivalent_post_b4c() {
        let src = include_str!("engine.rs");
        let body_start = src
            .find("fn worker_run(")
            .expect("H41: worker_run entry not found");
        let body_after = &src[body_start..];
        let body_end_off = body_after
            .find("\n// The worker thread for `LoadedModel::Qwen35` returns a sentinel error")
            .or_else(|| body_after.find("\n/// Worker-thread entry point"))
            .unwrap_or(body_after.len().min(200_000));
        let body = &body_after[..body_end_off];
        // The Request::Generate arm still calls `generate_once` for
        // Gemma 4 (the pre-C2c production path). Source-grep pin.
        assert!(
            body.contains("generate_once(g, &prompt_tokens, &params, registration.as_ref())"),
            "H41 FALSIFIED: post-B4c worker_run Gemma 4 Request::Generate \
             arm no longer routes through `generate_once`. SerialFifo \
             byte-equivalence with pre-B4c is BROKEN. The B4c label \
             refinement must NOT replace the existing forward call — \
             it only refines the typed-error string INSIDE the clamp."
        );
        // The Embed arm still calls `forward_embed_last` on Gemma 4
        // weights (pre-C2c production path).
        assert!(
            body.contains("g.weights.forward_embed_last(&prompt_tokens, &mut g.ctx)"),
            "H41 sanity: Embed Gemma 4 dispatch still routes through \
             `forward_embed_last` (pre-C2c surface). If this fails the \
             SerialFifo embed byte-equivalence is broken."
        );
        // The streaming arm still dispatches Gemma 4 through the
        // `LoadedModel::Gemma(g)` match arm post-clamp — verify the
        // arm structurally exists.
        assert!(
            body.contains("LoadedModel::Gemma(g) =>"),
            "H41 sanity: GenerateStream / Embed match arms still \
             dispatch on `LoadedModel::Gemma(g)`. If this fails the \
             entire Gemma 4 surface in worker_run has been gutted."
        );
    }

    /// **H42 (skip-mode pin)** — typed `MultiSeqError::Capability
    /// Unsupported` Display round-trip carries the iter-B4c-kernel
    /// label naming the deferred kernel surface AND preserves the
    /// existing iter-C2c-cont + B4c substrings (C2c surface
    /// preservation pin). Type-level + Display round-trip pin
    /// (Path B label refinement shape; mirrors C2d-cont H37).
    #[test]
    fn h42_capability_unsupported_label_names_iter_b4c_kernel_for_gemma4() {
        let err = MultiSeqError::CapabilityUnsupported {
            capability:
                "gemma4-forward-prefill-slot-N (iter-C2c-cont per ADR-040 §6.1.21 / iter-B4c-kernel per ADR-040 §6.1.25 — gated on B4c kernel slot-offset routing through src/serve/forward_prefill.rs + per-slot MultiSeqHbKvBuffers slot routing)",
        };
        let msg = format!("{}", err);
        assert!(
            msg.contains("gemma4-forward-prefill-slot-N"),
            "H42 FALSIFIED: typed-deferral label must name the deferred \
             capability (gemma4 forward path) for operator-actionable \
             diagnostics. Got: {msg}"
        );
        assert!(
            msg.contains("iter-B4c-kernel"),
            "H42 FALSIFIED: typed-deferral label must name the \
             implementing iter (iter-B4c-kernel) so operator log greps \
             land on the right pin pointer per ADR-040 §6.1.25. Got: {msg}"
        );
        assert!(
            msg.contains("iter-C2c-cont"),
            "H42 FALSIFIED: existing iter-C2c-cont prefix must be \
             PRESERVED — the C2c surface (H25 / C2d-cont H40 string-\
             match pins) is unchanged by B4c per the §6.1.25 path-B \
             label-refinement discipline. Got: {msg}"
        );
        assert!(
            msg.contains("forward_prefill.rs"),
            "H42 FALSIFIED: typed-deferral label must name the file \
             that needs the kernel work — Chesterton's fence on the \
             worker arm's string-prefix contract that handlers \
             string-match against. Got: {msg}"
        );
        assert!(
            msg.contains("MultiSeqHbKvBuffers"),
            "H42 FALSIFIED: typed-deferral label must name the gating \
             primitive (MultiSeqHbKvBuffers — the A3a sibling-struct \
             KV buffer that the kernel slot-offset routing must \
             consult); without this cite, a future iter that lifts \
             the deferral cannot grep for what unblocks it. Got: {msg}"
        );
    }

    /// **H43 (skip-mode, typed deferral label)** — pins that
    /// (a) all four worker arms carry the iter-B4c-kernel clamp string,
    /// (b) the `forward_prefill.rs` `slot_id` thread is NOT yet in
    ///     `worker_run` (deferral structural marker — once
    ///     iter-B4c-kernel lifts the kernel slot-offset routing, the
    ///     worker arm itself becomes load-bearing for `slot_id`
    ///     handoff to `forward_prefill_with_kv_cache_slot`).
    ///
    /// Defends the dual deferral discipline: the typed string surface
    /// (operator-facing) + the source-grep structural pin (reviewer-
    /// facing) move in lockstep. Mirrors C2d-cont H38.
    #[test]
    fn h43_typed_deferral_label_present_in_all_four_worker_arms_and_forward_prefill_slot_id_not_yet_threaded(
    ) {
        let src = include_str!("engine.rs");
        // Count Gemma 4 B4c clamp occurrences. Each of the 4 worker
        // arms (Generate / GenerateStream / Embed /
        // GenerateWithSoftTokens) should carry exactly one clamp
        // surfacing the iter-B4c-kernel deferral label.
        let clamp_label = "iter-B4c-kernel per ADR-040 §6.1.25";
        let n = src.matches(clamp_label).count();
        // The label appears in: 4 worker-arm clamps + this test
        // module's structural pins (the label and a comment-form).
        // Bound on the LOWER bound (at least 4 — the four worker-arm
        // clamps) so reviewer-facing test text doesn't double-count.
        assert!(
            n >= 4,
            "H43 FALSIFIED: expected at least 4 occurrences of the \
             iter-B4c-kernel label (one per worker arm: Generate, \
             GenerateStream, Embed, GenerateWithSoftTokens). Got {n}. \
             Drift here means the B4c label refinement is missing \
             from at least one of the four arms — partial coverage \
             breaks the deferral discipline."
        );

        // Pin: `forward_prefill_with_kv_cache_slot` (the
        // iter-B4c-kernel target API shape — a hypothetical sibling
        // of `forward_prefill_with_kv_cache` that accepts a
        // `slot_id: SlotId` parameter) is NOT yet called from
        // `worker_run`. When iter-B4c-kernel lands the kernel-side
        // routing, the worker arm must call the slot-aware variant
        // with `handle.slot_id` threaded through. Today this is
        // structurally absent — pin so a future iter that adds the
        // slot-aware call also removes this assertion.
        let body_start = src
            .find("fn worker_run(")
            .expect("H43: worker_run entry not found");
        let body_after = &src[body_start..];
        let body_end_off = body_after
            .find("\n// The worker thread for `LoadedModel::Qwen35` returns a sentinel error")
            .or_else(|| body_after.find("\n/// Worker-thread entry point"))
            .unwrap_or(body_after.len().min(200_000));
        let body = &body_after[..body_end_off];
        assert!(
            !body.contains("forward_prefill_with_kv_cache_slot"),
            "H43 FALSIFIED: worker_run now calls \
             `forward_prefill_with_kv_cache_slot` — this is the \
             iter-B4c-kernel kernel-slot-routing landing. Update H43 \
             to pin the call shape + remove this structural absence \
             assertion."
        );
        // Also pin: the worker_run body for Gemma 4 does NOT yet
        // thread `handle.slot_id` into any `forward_prefill*` call.
        // Sanity check via source-grep — `forward_prefill` appearances
        // in `worker_run` should NOT be followed by a `slot_id:` or
        // `, handle.slot_id` argument.
        //
        // We approximate with a negative match: today the worker
        // doesn't even mention `forward_prefill` by name (the call
        // happens inside `generate_once` / `generate_stream_once`).
        // If a future iter adds an inline `forward_prefill` call
        // with `handle.slot_id` threading, the assertion below
        // will trip and the test author must update both production
        // + this pin.
        assert!(
            !body.contains("forward_prefill_with_soft_tokens(&handle.slot_id"),
            "H43 FALSIFIED: worker_run now threads handle.slot_id \
             directly into `forward_prefill_with_soft_tokens` — this \
             is the iter-B4c-kernel landing. Update H43 to pin the \
             call shape + remove this structural absence assertion."
        );
    }

    /// **H44 (skip-mode)** — SlotAware + SlotId(0) for Gemma 4 routes
    /// through the existing forward path (NOT a SlotAware-only branch).
    /// Source-grep pin that the typed clamp is `matches!(loaded,
    /// LoadedModel::Gemma(_)) && handle.slot_id != SlotId(0)` (NOT
    /// `mode is SlotAware`).
    ///
    /// Rationale: under SlotAware with max_slots=N, SlotId(0) is the
    /// first slot handed out by InflightBatchedScheduler. We preserve
    /// byte-equivalence for SlotId(0) at SlotAware by keeping the
    /// existing forward path — only SlotId(N>0) trips the Path B
    /// clamp. This pin defends against a future drift that silently
    /// extends the clamp to "any SlotAware admission". Mirrors
    /// C2d-cont H39 for Gemma 4.
    #[test]
    fn h44_gemma4_clamp_is_slot_id_nonzero_only_not_mode_predicate() {
        let src = include_str!("engine.rs");
        let body_start = src
            .find("fn worker_run(")
            .expect("H44: worker_run entry not found");
        let body_after = &src[body_start..];
        let body_end_off = body_after
            .find("\n// The worker thread for `LoadedModel::Qwen35` returns a sentinel error")
            .or_else(|| body_after.find("\n/// Worker-thread entry point"))
            .unwrap_or(body_after.len().min(200_000));
        let body = &body_after[..body_end_off];
        // The Gemma 4 clamp predicate is `matches!(loaded,
        // LoadedModel::Gemma(_)) && handle.slot_id != SlotId(0)`.
        // Source-grep pin: this exact predicate must appear ≥4 times
        // (once per worker arm).
        let predicate = "matches!(loaded, LoadedModel::Gemma(_)) && handle.slot_id != SlotId(0)";
        let n = body.matches(predicate).count();
        assert!(
            n >= 4,
            "H44 FALSIFIED: expected the Gemma 4 clamp predicate \
             `{predicate}` in at least 4 worker arms. Got {n}. \
             Drift here may indicate the clamp extended to all \
             SlotAware admissions (breaking SlotId(0) byte-equivalence) \
             OR was removed from one of the four arms (incomplete \
             coverage)."
        );
    }

    /// **H45 (skip-mode, REVISED iter-C2d-cont-kernel iter-3 2026-05-30)** —
    /// Qwen35 + Qwen3VL worker arms not touched by B4c. Source-grep
    /// pin that the surviving Qwen35 clamp (post-iter-1 + post-iter-2 +
    /// post-iter-3 state: GenerateWithSoftTokens ONLY) carries its
    /// `iter-C2d-cont-kernel-iter-4` cite AND that no Qwen3VL clamp was
    /// accidentally added (Qwen3VL SlotAware activation is deferred
    /// to iter-C2e per §6.1.22 spawn arm).
    ///
    /// **Post-iter-1 (§6.1.27, 2026-05-29)** the Generate arm's
    /// `qwen35-forward-gpu-last-logits-slot-N` clamp label was REMOVED
    /// (the actual lift landed via `generate_qwen35_once_slot_aware`).
    /// **Post-iter-2 (§6.1.28, 2026-05-30)** the GenerateStream arm's
    /// `qwen35-forward-gpu-last-logits-slot-N-stream` clamp label was
    /// ALSO REMOVED (the actual lift landed via
    /// `generate_stream_qwen35_once_extended_slot_aware`).
    /// **Post-iter-3 (§6.1.29, 2026-05-30)** the Embed arm's
    /// `qwen35-forward-embed-last-slot-N` clamp label was ALSO REMOVED
    /// (the actual lift landed via `embed_qwen35_slot_aware`).
    /// **Post-iter-4 (§6.1.30, 2026-05-30)** the GenerateWithSoftTokens
    /// arm's `qwen35-forward-gpu-with-soft-tokens-slot-N` clamp label
    /// was ALSO REMOVED (the actual lift landed via
    /// `generate_qwen35_once_with_soft_tokens_slot_aware` +
    /// `generate_qwen35_once_with_soft_tokens_and_deepstack_slot_aware`).
    /// All four original C2d-cont labels are now gone — the Qwen35
    /// worker-arm lift arc is COMPLETE. H45's sibling-discipline intent
    /// (Qwen35 labels not touched by B4c) is preserved by pinning the
    /// `iter-C2d-cont-kernel` cite family (historical comments) +
    /// pinning iter-1/2/3/4 lift fns are all called (no B4c regression
    /// of Qwen35 lifts).
    ///
    /// Mirrors C2d-cont H40's sibling-discipline pin in reverse:
    /// where H40 pinned "Gemma 4 + Qwen3VL unchanged by C2d-cont",
    /// H45 pins "Qwen35 + Qwen3VL unchanged by B4c".
    #[test]
    fn h45_qwen35_and_qwen3vl_worker_arms_unchanged_by_b4c() {
        let src = include_str!("engine.rs");
        let body_start = src
            .find("fn worker_run(")
            .expect("H45: worker_run entry not found");
        let body_after = &src[body_start..];
        let body_end_off = body_after
            .find("\n// The worker thread for `LoadedModel::Qwen35` returns a sentinel error")
            .or_else(|| body_after.find("\n/// Worker-thread entry point"))
            .unwrap_or(body_after.len().min(200_000));
        let body = &body_after[..body_end_off];

        // Post-iter-4: NO Qwen35 worker-arm clamp labels remain. The
        // sibling-discipline intent ("Qwen35 not touched by B4c") is
        // preserved by pinning the surviving `iter-C2d-cont-kernel`
        // cite family in historical comments AND that all four
        // iter-1/2/3/4 lift fns are wired into worker_run (B4c didn't
        // accidentally remove any Qwen35 lift). The post-iter-3 H45
        // assertion that `qwen35-forward-gpu-with-soft-tokens-slot-N`
        // persisted reflected iter-3's state; iter-4 legitimately
        // removes that label too.

        // C2d-cont's iter-C2d-cont-kernel label family is preserved
        // (not accidentally rewritten by B4c to iter-B4c-kernel —
        // Qwen35's structural follow-up iter is iter-C2d-cont-kernel,
        // NOT B4c). Even post-iter-4 the historical comments at the
        // Generate / GenerateStream / Embed / SoftTokens lift forks
        // still reference iter-C2d-cont-kernel via the §6.1.27/28/29/30
        // cite chain.
        assert!(
            body.contains("iter-C2d-cont-kernel"),
            "H45 FALSIFIED: C2d-cont's iter-C2d-cont-kernel label \
             cite no longer present in worker_run. B4c must NOT \
             touch the Qwen35 clamp labels — Qwen35's follow-up iter \
             is iter-C2d-cont-kernel, not iter-B4c-kernel."
        );

        // iter-1/2/3/4 lift fns are all called (B4c must not regress
        // any of these — the sibling-discipline intent extended to the
        // post-iter-4 state).
        for lift_fn in [
            "super::engine_qwen35::generate_qwen35_once_slot_aware(",
            "super::engine_qwen35::generate_stream_qwen35_once_extended_slot_aware(",
            "super::engine_qwen35::embed_qwen35_slot_aware(",
            "super::engine_qwen35::generate_qwen35_once_with_soft_tokens_slot_aware(",
        ] {
            assert!(
                body.contains(lift_fn),
                "H45 FALSIFIED: iter-1/2/3/4 lift fn `{lift_fn}` is \
                 NOT called from worker_run. B4c must NOT regress any \
                 Qwen35 worker-arm lift (§6.1.27/28/29/30)."
            );
        }

        // ADR-040 iter-C2e (2026-05-30 §6.1.52) — Qwen3VL clamp
        // SHIPPED post-B4c (B4c label-refinement commit predates
        // C2e). Sibling discipline pin: B4c must not REMOVE the C2e
        // Qwen3VL clamp.
        assert!(
            body.contains(
                "matches!(loaded, LoadedModel::Qwen3VlText(_)) && handle.slot_id != SlotId(0)"
            ),
            "H45 FALSIFIED (post-C2e revision per §6.1.52): Qwen3VL \
             clamp missing from worker_run. iter-C2e SHIPPED 2026-05-30 \
             flipping the Qwen3VL SlotAware spawn arm to `Ok(Engine)` AND \
             adding the four worker-arm clamps; B4c must NOT regress the \
             C2e clamp."
        );
    }
}

// ---------------------------------------------------------------------------
// ADR-040 Phase E1 — production cutover decision + final ADR closure
// ceremony (2026-05-29).
//
// E1 is the FINAL closure block for ADR-040. The reopen trigger per
// ADR-040 §1.5 + §3.6 + §3.7 ("≥8 concurrent users sustained over 7
// days for any deployed instance, OR a customer ASKS for it
// explicitly") is NOT MET today. Per the decision matrix in §3.6, the
// production default REMAINS [`EngineMode::SerialFifo`]; SlotAware
// stays opt-in behind `--engine-mode=slot-aware` / `--scheduler
// inflight_batched` + `HF2Q_SCHEDULER=inflight_batched`. The
// kernel-level lifts (iter-A2b-cont, iter-C2d-cont-kernel,
// iter-B4c-kernel) survive as TYPED DEFERRALS pinned by H38 + H43
// label strings, ready to fire when the reopen trigger lands.
//
// H46–H50 are TDD source-grep pins over both `cli.rs` (for the
// operator-facing default + opt-in surface) and the ADR text (for the
// closure block's structural shape: ≥7 deferrals enumerated, AC
// status declared, reopen trigger framed).
// ---------------------------------------------------------------------------
/// ADR-005 iter-230 A1 — corpus helper for ADR-040 §6.1.x closure-block
/// doc-pins.
///
/// Commit `aeb6e87c` extracted the §6.1.x changelog from
/// `ADR-040-continuous-batching-reopen.md` (6141→1091 lines) into
/// `ADR-040-history.md`. The closure ceremonies those pins grep now live
/// in the HISTORY doc, so historical-closure assertions call this helper;
/// live-status/navigation assertions keep reading the MAIN doc directly
/// (deliberately NOT a concatenation — a marker moving out of the main
/// doc while a live section silently disappears must still fail the
/// live pins). `iter230_a1_historical_markers_exactly_once` (in
/// `adr040_phase_e1_closure_tests`) asserts every retargeted `### 6.1.x`
/// marker occurs exactly once in the history doc, closing the
/// duplicate-marker ambiguity.
#[cfg(test)]
pub(crate) fn adr040_history_doc() -> &'static str {
    include_str!("../../../docs/ADR-040-history.md")
}

#[cfg(test)]
mod adr040_phase_e1_closure_tests {
    use super::*;

    /// ADR-005 iter-230 AC-A1: every `### 6.1.x` marker that the
    /// retargeted doc-pins grep must occur EXACTLY ONCE in the history
    /// doc — a duplicate would make block extraction (`find` + next
    /// `### ` boundary) silently pick the wrong copy.
    #[test]
    fn iter230_a1_historical_markers_exactly_once() {
        let history = adr040_history_doc();
        for marker in [
            "### 6.1.26",
            "### 6.1.27",
            "### 6.1.30",
            "### 6.1.31",
            "### 6.1.32",
            "### 6.1.33",
            "### 6.1.34",
            "### 6.1.35",
            "### 6.1.36",
            "### 6.1.37",
            "### 6.1.38",
            "### 6.1.39",
            "### 6.1.40",
            "### 6.1.41",
            "### 6.1.42",
            "### 6.1.43",
            "### 6.1.44",
            "### 6.1.45",
            "### 6.1.52",
            "### 6.1.55",
        ] {
            let n = history.matches(marker).count();
            assert_eq!(
                n, 1,
                "ADR-040-history.md must contain {marker:?} exactly once, found {n}"
            );
        }
    }

    /// **H46 (skip-mode)** — `EngineMode::default()` is `SerialFifo`
    /// (production default unchanged by Phase E1 closure).
    ///
    /// This pin is BOTH a behavioural assertion (the impl returns the
    /// SerialFifo variant) AND a source-grep assertion (the
    /// `impl Default for EngineMode` block names `Self::SerialFifo`
    /// as the body). Drift in either form would mean the E1 decision
    /// matrix in §3.6 was silently overridden — the reopen trigger is
    /// not met today, so the cutover MUST NOT have fired.
    #[test]
    fn h46_engine_mode_default_is_serial_fifo_per_e1_decision() {
        // Behavioural half: Default::default() returns SerialFifo.
        let mode = EngineMode::default();
        assert!(
            matches!(mode, EngineMode::SerialFifo),
            "H46 FALSIFIED: EngineMode::default() returned {mode:?} \
             — Phase E1 §3.6 decision is KEEP SerialFifo until the \
             reopen trigger (≥8 concurrent OR customer ask) fires. \
             A non-SerialFifo default means the cutover landed without \
             the gate."
        );

        // Source-grep half: the impl block names Self::SerialFifo.
        // Drift defence — catches a future edit that flips the
        // default via clever indirection (e.g. `Self::SlotAware {
        // max_slots: 1 }` which happens to admit identical
        // single-slot semantics but breaks ADR-040 §3.6 byte-equivalence
        // expectations).
        let src = include_str!("engine.rs");
        let default_marker = "impl Default for EngineMode";
        let idx = src
            .find(default_marker)
            .expect("H46: `impl Default for EngineMode` block not found");
        // Restrict the window to a small region after the marker so
        // we don't accidentally match a sibling `impl Default` that
        // appears later in the file.
        let window = &src[idx..idx + 600.min(src.len() - idx)];
        assert!(
            window.contains("Self::SerialFifo"),
            "H46 FALSIFIED: `impl Default for EngineMode` no longer \
             names `Self::SerialFifo` in its body. Phase E1 §3.6 \
             decision pins this as the production default until the \
             reopen trigger fires."
        );
    }

    /// **H47 (skip-mode)** — SlotAware is opt-in via BOTH `--scheduler
    /// inflight_batched` (CLI flag, per §6.1.9 C4) and
    /// `HF2Q_SCHEDULER=inflight_batched` (env, per §6.1.9 C4).
    /// Source-grep over `cli.rs` for the flag declaration + over
    /// `serve/mod.rs` for the env wiring.
    ///
    /// This is the operator-facing forward runbook: when the reopen
    /// trigger fires, the operator does NOT need a new release — the
    /// opt-in surface is already there. Drift here would mean the
    /// runbook is broken before the trigger lands.
    #[test]
    fn h47_slot_aware_is_opt_in_via_cli_flag_and_env_per_c4() {
        // CLI flag — `cli.rs` declares the `--scheduler` flag with
        // a `SchedulerArg` value enum + the `--max-slots` companion
        // flag.
        let cli_src = include_str!("../../cli.rs");
        assert!(
            cli_src.contains("--scheduler") || cli_src.contains("\"scheduler\""),
            "H47 FALSIFIED: `cli.rs` no longer declares the \
             `--scheduler` CLI flag. Operators have no opt-in path \
             for SlotAware — Phase E1 forward runbook is broken."
        );
        assert!(
            cli_src.contains("SchedulerArg"),
            "H47 FALSIFIED: `cli.rs` no longer declares the \
             `SchedulerArg` clap ValueEnum. The opt-in flag's value \
             discipline is gone."
        );
        assert!(
            cli_src.contains("--max-slots") || cli_src.contains("\"max-slots\""),
            "H47 FALSIFIED: `cli.rs` no longer declares the \
             `--max-slots` CLI flag (§3.4 default = 4 under
             InflightBatched)."
        );
        assert!(
            cli_src.contains("InflightBatched"),
            "H47 FALSIFIED: `cli.rs` no longer names the \
             `InflightBatched` SchedulerArg variant — the opt-in \
             discriminant is gone."
        );

        // Env wiring — `serve/mod.rs` reads `HF2Q_SCHEDULER` +
        // `HF2Q_MAX_SLOTS` via `parse_scheduler_config`.
        let mod_src = include_str!("../mod.rs");
        assert!(
            mod_src.contains("HF2Q_SCHEDULER"),
            "H47 FALSIFIED: `serve/mod.rs` no longer reads the \
             `HF2Q_SCHEDULER` env var — env-side opt-in is gone."
        );
        assert!(
            mod_src.contains("HF2Q_MAX_SLOTS"),
            "H47 FALSIFIED: `serve/mod.rs` no longer reads the \
             `HF2Q_MAX_SLOTS` env var — operator can't tune slot \
             count via env."
        );
        assert!(
            mod_src.contains("parse_scheduler_config"),
            "H47 FALSIFIED: `serve/mod.rs` no longer threads \
             `parse_scheduler_config` — the CLI + env join point \
             is gone."
        );
    }

    /// **H48 (skip-mode)** — The E1 closure block (§6.1.26) honestly
    /// enumerates ≥7 surviving typed deferrals, each with an
    /// operator-grep'able iter-N label.
    ///
    /// The enumeration is the operator's forward runbook: when the
    /// reopen trigger fires, these are the iters that must land
    /// before SlotAware end-to-end byte-equivalence with SerialFifo
    /// is provable. Hiding a deferral here would silently shrink the
    /// runbook.
    #[test]
    fn h48_e1_closure_enumerates_at_least_7_typed_deferrals() {
        let adr = crate::serve::api::engine::adr040_history_doc() /* iter-230 A1: §6.1.x moved to history (aeb6e87c) */;
        let closure_marker = "### 6.1.26";
        let closure_start = adr.find(closure_marker).expect(
            "H48: ADR §6.1.26 closure block not found — E1 \
                     closure ceremony has not landed",
        );
        let closure_end_off = adr[closure_start..]
            .find("\n---\n")
            .or_else(|| adr[closure_start..].find("\n## "))
            .unwrap_or_else(|| adr[closure_start..].len().min(40_000));
        let closure_body = &adr[closure_start..closure_start + closure_end_off];

        // Enumerate the typed deferrals that must survive past E1.
        let required_deferrals = [
            "iter-A2b-cont",        // forward-path linear-attn dispatch
            "iter-C2d-cont-kernel", // Qwen35 worker hot path lift
            "iter-B4c-kernel",      // Gemma 4 forward_prefill slot lift
            "iter-A2c",             // fork_seq cross-slot kernel
            "iter-A3c",             // Gemma 4 fork_seq cross-slot
            "iter-A3b-2",           // DenseKvBuffers full lift
            "iter-A3b-3",           // MlxKvCache full lift
        ];

        let mut missing = Vec::new();
        for label in &required_deferrals {
            if !closure_body.contains(label) {
                missing.push(*label);
            }
        }
        assert!(
            missing.is_empty(),
            "H48 FALSIFIED: §6.1.26 closure block omits {} required \
             typed-deferral label(s): {:?}. Each surviving deferral \
             MUST be named in the closure for the operator runbook \
             to be complete. ADR-040 §7 mantra (\"no fallback, no \
             stub\") demands every deferral carry an operator- \
             grep'able iter-N label.",
            missing.len(),
            missing
        );

        // Defence-in-depth: count the total number of distinct
        // `iter-` labels named in the closure as a coarse upper-bound
        // sanity check. The 7 required labels above are the
        // minimum; the closure may name more.
        let total_iter_mentions = closure_body.matches("iter-").count();
        assert!(
            total_iter_mentions >= 7,
            "H48 FALSIFIED: closure block names only \
             {total_iter_mentions} `iter-*` references in total. \
             Need at least 7 for the deferral enumeration to be \
             complete."
        );
    }

    /// **H49 (skip-mode)** — The E1 closure block declares AC-1..AC-5
    /// status (MET / DEFERRED / WAIVED) per ADR §5.
    ///
    /// Each AC must have an explicit status verdict in the closure
    /// so the operator + future-iter author can answer "is the
    /// reopen trigger satisfied today?" with a single grep.
    #[test]
    fn h49_e1_closure_declares_ac_status_for_each_acceptance_criterion() {
        let adr = crate::serve::api::engine::adr040_history_doc() /* iter-230 A1: §6.1.x moved to history (aeb6e87c) */;
        let closure_marker = "### 6.1.26";
        let closure_start = adr
            .find(closure_marker)
            .expect("H49: ADR §6.1.26 closure block not found");
        let closure_end_off = adr[closure_start..]
            .find("\n---\n")
            .or_else(|| adr[closure_start..].find("\n## "))
            .unwrap_or_else(|| adr[closure_start..].len().min(40_000));
        let closure_body = &adr[closure_start..closure_start + closure_end_off];

        // ADR §5 declares AC-1..AC-5. Each must be named + carry
        // a status verdict (MET / DEFERRED / WAIVED).
        let required_acs = ["AC-1", "AC-2", "AC-3", "AC-4", "AC-5"];
        for ac in &required_acs {
            assert!(
                closure_body.contains(ac),
                "H49 FALSIFIED: §6.1.26 closure block does not name \
                 `{ac}`. ADR §5 declares AC-1..AC-5; each must have \
                 a status verdict (MET / DEFERRED / WAIVED) in the \
                 final closure for operator reading."
            );
        }

        // Status verdict vocabulary must appear in the AC section.
        let met_count = closure_body.matches("MET").count();
        let deferred_count = closure_body.matches("DEFERRED").count();
        assert!(
            met_count + deferred_count >= 5,
            "H49 FALSIFIED: §6.1.26 closure block names \
             {met_count} `MET` + {deferred_count} `DEFERRED` \
             verdicts. Need at least 5 total to cover AC-1..AC-5 \
             (each MUST have an explicit status declaration)."
        );
    }

    /// **H50 (skip-mode)** — The E1 closure block explicitly states
    /// the reopen trigger is NOT MET today AND names what would
    /// trigger re-opening (customer ask OR ≥8 concurrent sustained).
    ///
    /// This is the load-bearing decision pin: anyone reading §6.1.26
    /// must immediately see why SerialFifo stayed the default + what
    /// fires the next iter.
    #[test]
    fn h50_e1_closure_documents_reopen_trigger_status() {
        let adr = crate::serve::api::engine::adr040_history_doc() /* iter-230 A1: §6.1.x moved to history (aeb6e87c) */;
        let closure_marker = "### 6.1.26";
        let closure_start = adr
            .find(closure_marker)
            .expect("H50: ADR §6.1.26 closure block not found");
        let closure_end_off = adr[closure_start..]
            .find("\n---\n")
            .or_else(|| adr[closure_start..].find("\n## "))
            .unwrap_or_else(|| adr[closure_start..].len().min(40_000));
        let closure_body = &adr[closure_start..closure_start + closure_end_off];

        // Decision pin: KEEP SerialFifo must appear verbatim.
        assert!(
            closure_body.contains("KEEP SerialFifo") || closure_body.contains("Keep SerialFifo"),
            "H50 FALSIFIED: §6.1.26 closure block does not state \
             the decision `KEEP SerialFifo`. The §3.6 decision matrix \
             requires an explicit verdict; ambiguity here means the \
             cutover status is unclear to operators."
        );

        // Reopen trigger status — explicitly "NOT MET".
        let not_met_present = closure_body.contains("NOT MET")
            || closure_body.contains("not met")
            || closure_body.contains("not yet met");
        assert!(
            not_met_present,
            "H50 FALSIFIED: §6.1.26 closure block does not state the \
             reopen trigger is NOT MET today. ADR-040 §1.5 + §3.6 \
             require an explicit status declaration."
        );

        // Trigger conditions named — customer ask OR ≥8 concurrent.
        let customer_named = closure_body.contains("customer") || closure_body.contains("Customer");
        assert!(
            customer_named,
            "H50 FALSIFIED: §6.1.26 closure block does not name \
             `customer` as one of the reopen-trigger conditions. \
             ADR-005 + ADR-040 §3.7 cite \"customer asks explicitly\" \
             as one branch of the trigger."
        );
        let concurrency_named = closure_body.contains("≥8")
            || closure_body.contains(">=8")
            || closure_body.contains("8 concurrent");
        assert!(
            concurrency_named,
            "H50 FALSIFIED: §6.1.26 closure block does not name the \
             ≥8 concurrent users threshold (ADR-005 reopen-trigger \
             condition cited verbatim in ADR-040 §1.5)."
        );
    }

    /// **Status marker pin** — ADR-040 top-of-document Status line.
    ///
    /// Era history: E1 closure marked the ADR `CLOSED` (§6.1.26); Phase
    /// F REOPENED it 2026-06-24 when the throughput bench falsified the
    /// closure (0.85× regression, §0); the REOPENED era legitimately
    /// ENDED at `dc927f39` when the target workload was served
    /// (coherence + capacity + speed bars met — §0 milestone ledger).
    /// ADR-005 iter-230 A1 retargeted this pin from the expired
    /// `REOPENED` literal to the earned live status. Three invariants:
    /// (1) the main doc's first Status line carries the earned state,
    /// (2) the main doc still links the extracted history doc
    /// (navigability after the aeb6e87c split), (3) the history doc
    /// retains the REOPENED-era record (the era must stay auditable,
    /// not vanish with the status flip).
    #[test]
    fn adr040_status_line_carries_earned_live_status() {
        let adr = include_str!("../../../docs/ADR-040-continuous-batching-reopen.md");
        // The Status line is the first `- **Status**:` line in the
        // file (per ADR-040 header). Restrict scan to the first
        // 4 KB so we don't accidentally match a §6.1.* status mention
        // deeper in the document.
        let header = &adr[..4096.min(adr.len())];
        let status_idx = header
            .find("- **Status**:")
            .expect("status: top-of-document Status line not found");
        let status_line_end = header[status_idx..]
            .find('\n')
            .unwrap_or(header.len() - status_idx);
        let status_line = &header[status_idx..status_idx + status_line_end];
        assert!(
            status_line.contains("TARGET WORKLOAD SERVED"),
            "FALSIFIED: ADR-040 top-of-document Status line does not \
             carry the earned `TARGET WORKLOAD SERVED` status. If the \
             status legitimately changed again, retarget this pin WITH \
             a doc-comment era note (as iter-230 A1 did for REOPENED); \
             do not delete it. Line was: {status_line:?}"
        );
        assert!(
            adr.contains("ADR-040-history.md"),
            "FALSIFIED: main ADR-040 doc no longer references \
             ADR-040-history.md — the extracted §6.1.x changelog must \
             stay navigable from the live doc."
        );
        // The era record must stay auditable across BOTH docs: the main
        // doc keeps the literal `REOPENED` era mentions; the history doc
        // keeps the SPECIFIC reopen-trigger record (§6.1.26's H50-pinned
        // block), not just an incidental substring.
        assert!(
            adr.contains("REOPENED"),
            "FALSIFIED: main ADR-040 doc lost its REOPENED-era \
             mentions — the Phase F reopen must remain auditable."
        );
        let history = crate::serve::api::engine::adr040_history_doc();
        assert!(
            history.contains("Reopen-trigger status (per ADR-040 §1.5 + §3.7)")
                && history.contains("Reopen trigger NOT MET today."),
            "FALSIFIED: ADR-040-history.md lost the reopen-trigger \
             record (the §6.1.26-era block H50 pins) — the Phase F \
             reopen must remain auditable."
        );
    }
}

// ---------------------------------------------------------------------------
// ADR-040 iter-B4c-kernel iter-1 (2026-05-30) — Gemma 4 worker hot path
// Generate-arm lift onto persistent multi-seq per-layer
// `MultiSeqHbKvBuffers` scaffold (Gemma 4 mirror of Qwen35
// iter-C2d-cont-kernel iter-1 per §6.1.27).
//
// Path B (iter-1 = Generate-arm-only scaffold lift, iter-{2,3,4,5,LCP,G} =
// typed sub-deferrals) chosen over Path A (full 4-arm + kernel-forward
// lift in one iter) on three risk-symmetry grounds mirroring §6.1.27:
// 1. Kernel-prerequisite gap: Gemma 4 has NO equivalent of Qwen35's
//    B4b decode-path slot threading (per §6.1.20).  `forward_prefill.rs`
//    / `forward_prefill_with_soft_tokens` / `forward_embed_last` have
//    no `slot_id` parameter; `grep slot_id src/serve/forward_prefill.rs`
//    returns 0 hits.  The kernel-forward slot routing IS the iter-2 work.
// 2. Surface area for the kernel step alone exceeds 600 LOC (per §6.1.25
//    Path A risk analysis: 30 layers × 3 KV variants × xlen optional ×
//    3 inline alloc sites — `forward_prefill.rs:843-882`,
//    `forward_prefill_batched.rs:443-475`, `forward_gpu.rs:443-459`).
// 3. H1/H2/H23/H44 byte-equivalence pin contract preserved via the
//    `slot_id != SlotId(0)` predicate (H77 source-grep pin); the lift
//    fork is unreachable from SerialFifo (always SlotId(0)) and from
//    SlotAware + SlotId(0) (first-slot byte-equivalence with SerialFifo).
//
// What iter-1 ships (load-bearing primitives for iter-{2,3,4,5}):
//   * `MultiSeqHbKvBuffers::reset_for_slot(slot: SlotId)` + the sibling
//     `MultiSeqHybridKvBuffers::reset_for_slot(slot: SlotId)` (per-slot
//     cursor reset primitives — cross-architecture mirror of Qwen35
//     `HybridKvCache::reset_for_slot` per §6.1.27).
//   * `engine::generate_gemma4_once_slot_aware(g, .., &mut multi_seq_kv,
//     slot_id) -> Result<GenerationResult>` orchestrator scaffold —
//     bounds-checks slot_id + entry reset_for_slot + typed-deferred
//     kernel-forward step (iter-B4c-kernel-iter-2) + exit reset_for_slot.
//   * `worker_run` Gemma 4 Generate arm: clamp REPLACED with lift fork
//     (take-and-restore borrow pattern on `g.multi_seq_kv`).
//   * 3 remaining Gemma 4 worker arms (GenerateStream / Embed /
//     GenerateWithSoftTokens) RELABELED with
//     `iter-B4c-kernel-iter-{3,4,5} per ADR-040 §6.1.31` cites (existing
//     C2c §6.1.21 + B4c §6.1.25 prefixes preserved verbatim so
//     H42 / H25 / H40 / H43 string-match pins keep passing).
//
// Tests (H77-H83 mirror H51-H57 from iter-C2d-cont-kernel iter-1 1:1
// across the Qwen35 → Gemma 4 boundary):
//   H77 (skip-mode): SlotId(0) Gemma 4 routes through existing
//                    generate_once dispatch byte-equivalent.
//   H78 (skip-mode): iter-B4c-kernel iter-1 lift landed for Gemma 4
//                    Generate arm.
//   H79 (skip-mode): lift call site takes + restores g.multi_seq_kv.
//   H80 (skip-mode): orchestrator calls reset_for_slot at entry + exit
//                    across every per-layer buffer.
//   H81 (skip-mode): lift handles g.multi_seq_kv == None with typed
//                    error (defense-in-depth; impossible at runtime per
//                    C2c spawn invariant but pinned).
//   H82 (skip-mode): Qwen35 + Qwen3VL + Gemma 4 GenerateStream / Embed /
//                    GenerateWithSoftTokens UNCHANGED by iter-1.
//   H83 (skip-mode): iter-1 sub-deferrals named for remaining 3 Gemma 4
//                    arms (iter-B4c-kernel-iter-{3,4,5}) AND for the
//                    kernel-forward step itself (iter-B4c-kernel-iter-2).
// ---------------------------------------------------------------------------

#[cfg(test)]
mod adr040_phase_b_iter_b4c_kernel_iter1_gemma4_tests {
    // Skip-mode source-grep tests; intentionally NO `use super::*;`
    // (tests rely on `include_str!` against engine.rs + the ADR doc).

    /// **H77 (skip-mode)** — SlotId(0) Gemma 4 routes through the
    /// existing `generate_once` Generate-arm dispatch.  Source-grep
    /// pin: post-iter-1, the worker arm STILL contains the
    /// `generate_once(g, &prompt_tokens, &params, registration.as_ref())`
    /// call AT the bottom of the Generate arm's `match &mut loaded`
    /// dispatch block AND the predicate guarding the lift is
    /// `matches!(loaded, LoadedModel::Gemma(_)) && handle.slot_id != SlotId(0)`.
    ///
    /// Both SerialFifo (`FifoSchedulerAdapter` always hands out
    /// SlotId(0)) AND SlotAware + SlotId(0) (first-slot pin) short-
    /// circuit BELOW the lift fork and hit the existing dispatch
    /// verbatim — preserves H1/H2/H23/H41/H44 byte-equivalence chain.
    ///
    /// Mirrors iter-C2d-cont-kernel iter-1 H51 for the Gemma 4 surface.
    #[test]
    fn h77_slot_id_0_gemma4_routes_through_generate_once_byte_equivalent() {
        let src = include_str!("engine.rs");
        let body_start = src
            .find("fn worker_run(")
            .expect("H77: worker_run entry not found");
        let body_after = &src[body_start..];
        let body_end_off = body_after
            .find("\n// The worker thread for `LoadedModel::Qwen35` returns a sentinel error")
            .or_else(|| body_after.find("\n/// Worker-thread entry point"))
            .unwrap_or(body_after.len().min(200_000));
        let body = &body_after[..body_end_off];
        // The Request::Generate arm still calls `generate_once` for
        // Gemma 4 at the SlotId(0) path (bottom of the match arm).
        // Source-grep pin.
        assert!(
            body.contains("generate_once(g, &prompt_tokens, &params, registration.as_ref())"),
            "H77 FALSIFIED: post-iter-1 worker_run Gemma 4 Request::Generate \
             SlotId(0) arm no longer routes through `generate_once`. \
             SerialFifo + SlotId(0) byte-equivalence is BROKEN. \
             iter-1's lift must be SLOT-N>0 ONLY."
        );
        // The lift fork predicate `slot_id != SlotId(0)` exists in the
        // Gemma 4 Generate arm — the SLOT-N>0 path takes the lift.
        let predicate = "matches!(loaded, LoadedModel::Gemma(_)) && handle.slot_id != SlotId(0)";
        let n = body.matches(predicate).count();
        assert!(
            n >= 4,
            "H77 FALSIFIED: expected the Gemma 4 lift-fork predicate \
             `{predicate}` in at least 4 worker arms (Generate now uses \
             it for the lift; GenerateStream / Embed / SoftTokens still \
             use it for the clamp). Got {n}. \
             Drift here may indicate the predicate was extended to all \
             SlotAware admissions (breaking SlotId(0) byte-equivalence)."
        );
    }

    /// **H78 (skip-mode)** — iter-1 lift IS landed for the Gemma 4
    /// Generate arm.  Source-grep pin that the new orchestrator
    /// `generate_gemma4_once_slot_aware` IS called from `worker_run`'s
    /// Gemma 4 Generate arm AND the call site passes `slot_id` (the
    /// admit'd handle's `SlotId`) instead of a hard-coded SlotId(0).
    ///
    /// Mirrors iter-C2d-cont-kernel iter-1 H52.
    #[test]
    fn h78_iter1_lift_landed_for_gemma4_generate_arm() {
        let src = include_str!("engine.rs");
        // The slot-aware orchestrator fn is defined in this file.
        assert!(
            src.contains("fn generate_gemma4_once_slot_aware("),
            "H78 FALSIFIED: `generate_gemma4_once_slot_aware` is NOT \
             defined in engine.rs. iter-B4c-kernel iter-1 production \
             surface MISSING — orchestrator scaffold not landed."
        );
        // The orchestrator is CALLED from worker_run.
        let body_start = src
            .find("fn worker_run(")
            .expect("H78: worker_run entry not found");
        let body_after = &src[body_start..];
        let body_end_off = body_after
            .find("\n// The worker thread for `LoadedModel::Qwen35` returns a sentinel error")
            .or_else(|| body_after.find("\n/// Worker-thread entry point"))
            .unwrap_or(body_after.len().min(200_000));
        let body = &body_after[..body_end_off];
        assert!(
            body.contains("generate_gemma4_once_slot_aware("),
            "H78 FALSIFIED: worker_run does NOT call \
             `generate_gemma4_once_slot_aware`. iter-1 lift is not \
             wired into the dispatch fork."
        );
        // The lift call passes `slot_id` (the unbound captured SlotId
        // local) rather than a hard-coded SlotId(0) literal — the
        // load-bearing per-slot routing witness.
        let call_marker = "generate_gemma4_once_slot_aware(";
        let call_idx = body
            .find(call_marker)
            .expect("call_marker present (asserted above)");
        let call_window = &body[call_idx..(call_idx + 800).min(body.len())];
        assert!(
            call_window.contains("slot_id"),
            "H78 FALSIFIED: `generate_gemma4_once_slot_aware` call site \
             does not pass `slot_id`. The orchestrator must receive the \
             admit'd handle's SlotId, not a hard-coded SlotId(0)."
        );
        assert!(
            !call_window.contains("SlotId(0)"),
            "H78 FALSIFIED: `generate_gemma4_once_slot_aware` call site \
             contains a hard-coded `SlotId(0)` literal. Per-slot \
             routing is broken — the orchestrator must receive the \
             admit'd handle's SlotId verbatim."
        );
    }

    /// **H79 (skip-mode)** — lift call site takes + restores
    /// `g.multi_seq_kv`.  The take-and-restore borrow pattern is the
    /// load-bearing primitive that resolves the partial-borrow conflict
    /// between `&mut g.multi_seq_kv` and the dense `&mut g.lcp_registry`
    /// / `&mut g.prompt_cache` accesses inside the orchestrator.
    ///
    /// Pin defends two regressions:
    /// (a) `take()` but no put-back → next request fails the C2c
    ///     spawn-arm invariant (multi_seq_kv.is_some()).
    /// (b) Clone instead of take → cross-request KV state isolation
    ///     breaks (slot N's bytes leak into slot M's view).
    ///
    /// Mirrors iter-C2d-cont-kernel iter-1 H53.
    #[test]
    fn h79_lift_call_site_takes_and_restores_g_multi_seq_kv() {
        let src = include_str!("engine.rs");
        let body_start = src
            .find("fn worker_run(")
            .expect("H79: worker_run entry not found");
        let body_after = &src[body_start..];
        let body_end_off = body_after
            .find("\n// The worker thread for `LoadedModel::Qwen35` returns a sentinel error")
            .or_else(|| body_after.find("\n/// Worker-thread entry point"))
            .unwrap_or(body_after.len().min(200_000));
        let body = &body_after[..body_end_off];
        // The take pattern.
        assert!(
            body.contains("g.multi_seq_kv.take()"),
            "H79 FALSIFIED: lift call site does NOT take `g.multi_seq_kv`. \
             Partial-borrow conflict with `&mut g.lcp_registry` / \
             `&mut g.prompt_cache` will surface as a compile-time error \
             OR the lift will silently clone, breaking per-slot \
             isolation."
        );
        // The put-back pattern.
        assert!(
            body.contains("g.multi_seq_kv = Some(multi_seq);"),
            "H79 FALSIFIED: lift call site does NOT restore \
             `g.multi_seq_kv` after the call. The C2c spawn-arm \
             invariant (`multi_seq_kv.is_some()` for SlotAware Gemma 4) \
             will be violated on the next request."
        );
    }

    /// **H80 (skip-mode)** — `MultiSeqHbKvBuffers::reset_for_slot` is
    /// defined AND the orchestrator body calls it at entry + exit
    /// across EVERY per-layer buffer.  Cross-request state isolation
    /// within the slot.
    ///
    /// Mirrors iter-C2d-cont-kernel iter-1 H54 for the Gemma 4 surface
    /// (the iter-1 ADD of the `reset_for_slot` primitive on the
    /// `MultiSeqHbKvBuffers` + sibling `MultiSeqHybridKvBuffers` types).
    #[test]
    fn h80_reset_for_slot_called_at_entry_and_exit_per_layer() {
        let src = include_str!("engine.rs");
        // The `reset_for_slot` primitive is defined in
        // `gemma4/kv_cache.rs` — verify via cross-file source-grep
        // that the production callsite knows the name.
        let kv_src = include_str!("../../../src/inference/models/gemma4/kv_cache.rs");
        assert!(
            kv_src.contains("pub fn reset_for_slot("),
            "H80 FALSIFIED: `reset_for_slot` is NOT defined in \
             gemma4/kv_cache.rs. iter-1's load-bearing primitive is \
             missing."
        );
        // The orchestrator body calls `reset_for_slot` at LEAST TWICE
        // (entry + exit) on the per-layer buffers via `buf.reset_for_slot(slot_id)`.
        let fn_marker = "fn generate_gemma4_once_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H80: generate_gemma4_once_slot_aware not found");
        // Take a generous window (~10 KB) to cover the fn body.
        let fn_window = &src[fn_idx..(fn_idx + 10_000).min(src.len())];
        let n = fn_window.matches("reset_for_slot(slot_id)").count();
        assert!(
            n >= 2,
            "H80 FALSIFIED: `generate_gemma4_once_slot_aware` body \
             contains `reset_for_slot(slot_id)` only {n} times; \
             expected >= 2 (entry + exit). Cross-request slot \
             isolation is BROKEN."
        );
        // The orchestrator iterates per-layer (`multi_seq_kv.iter_mut`)
        // so reset_for_slot is called for every layer entry.
        assert!(
            fn_window.contains("multi_seq_kv.iter_mut()"),
            "H80 FALSIFIED: orchestrator does not iterate per-layer \
             via `multi_seq_kv.iter_mut()`. Per-layer reset coverage \
             is incomplete — only the first layer's slot would be \
             reset."
        );
    }

    /// **H81 (skip-mode)** — lift handles `g.multi_seq_kv.is_none()`
    /// with a typed error (defense-in-depth).  Impossible at runtime
    /// per the C2c spawn-arm invariant, but the worker arm surfaces a
    /// typed `capability_unsupported:` anyhow error with operator-
    /// grep'able label `"iter-B4c-kernel iter-1"` + `"multi_seq_kv is None"`
    /// substring instead of panicking.
    ///
    /// Mirrors iter-C2d-cont-kernel iter-1 H55.
    #[test]
    fn h81_lift_handles_multi_seq_kv_none_with_typed_error() {
        let src = include_str!("engine.rs");
        let body_start = src
            .find("fn worker_run(")
            .expect("H81: worker_run entry not found");
        let body_after = &src[body_start..];
        let body_end_off = body_after
            .find("\n// The worker thread for `LoadedModel::Qwen35` returns a sentinel error")
            .or_else(|| body_after.find("\n/// Worker-thread entry point"))
            .unwrap_or(body_after.len().min(200_000));
        let body = &body_after[..body_end_off];
        // Defense-in-depth: the None branch surfaces a typed error
        // with the iter cite + variable identification.
        assert!(
            body.contains("multi_seq_kv is None at SlotId("),
            "H81 FALSIFIED: lift `None` branch does NOT surface the \
             operator-grep'able `multi_seq_kv is None at SlotId(...)` \
             substring. Defense-in-depth typed error missing."
        );
        assert!(
            body.contains("iter-B4c-kernel iter-1"),
            "H81 FALSIFIED: lift error message does NOT name the \
             implementing iter (`iter-B4c-kernel iter-1`). Operator \
             log greps cannot land on the right pin pointer."
        );
        assert!(
            body.contains("provision_multi_seq_kv_for_slot_aware"),
            "H81 FALSIFIED: lift error message does NOT name the \
             spawn-time provisioning fn the C2c invariant relies on. \
             Operator cannot trace the invariant violation back to \
             the spawn-arm wiring without this cite."
        );
        // NO panic / unwrap on the option at the lift call site —
        // verified by ensuring the path takes the explicit `match`
        // arm via the `take()` body (the take returns Option, the
        // match unwraps via Some / None branches).
        let take_idx = body
            .find("let mut multi_seq = match g.multi_seq_kv.take()")
            .expect("H81: take-and-match shape not found");
        let take_window = &body[take_idx..(take_idx + 2000).min(body.len())];
        assert!(
            take_window.contains("None =>"),
            "H81 FALSIFIED: take match arm does not cover the `None =>` \
             branch explicitly. The lift may panic on the impossible \
             None state."
        );
        assert!(
            !take_window.contains(".unwrap()"),
            "H81 FALSIFIED: take call site uses `.unwrap()` on the \
             multi_seq_kv option. Defense-in-depth typed error path \
             is bypassed."
        );
    }

    /// **H82 (skip-mode, REVISED iter-B4c-kernel iter-3 2026-05-30)** —
    /// Qwen35 + Qwen3VL UNCHANGED by Gemma 4 iter-1 AND the remaining
    /// Gemma 4 worker arms (Embed + GenerateWithSoftTokens) still
    /// carry their C2c clamps.
    ///
    /// Sibling-discipline pin: iter-1 lifts ONLY the Gemma 4 Generate
    /// arm; the 3 other arms are typed sub-deferrals (iter-B4c-kernel-
    /// iter-{3,4,5}).  Drift here means iter-1 accidentally touched a
    /// surface it should not have.
    ///
    /// **Post-iter-3 (§6.1.35, 2026-05-30)** the GenerateStream arm's
    /// `gemma4-forward-prefill-slot-N` clamp label was REMOVED (the
    /// actual lift landed via `generate_stream_gemma4_once_slot_aware`).
    /// H82's sibling-discipline intent is preserved by pinning the
    /// SURVIVING Embed + GenerateWithSoftTokens C2c clamps + the
    /// iter-3 lift fn being called.
    ///
    /// Mirrors iter-C2d-cont-kernel iter-1 H56.
    #[test]
    fn h82_qwen35_qwen3vl_and_other_gemma4_arms_unchanged_by_iter1() {
        let src = include_str!("engine.rs");
        let body_start = src
            .find("fn worker_run(")
            .expect("H82: worker_run entry not found");
        let body_after = &src[body_start..];
        let body_end_off = body_after
            .find("\n// The worker thread for `LoadedModel::Qwen35` returns a sentinel error")
            .or_else(|| body_after.find("\n/// Worker-thread entry point"))
            .unwrap_or(body_after.len().min(200_000));
        let body = &body_after[..body_end_off];

        // Qwen35 worker-arm lift fns from iter-C2d-cont-kernel
        // iter-1/2/3/4 (§6.1.27-30) all STILL called from worker_run.
        for lift_fn in [
            "super::engine_qwen35::generate_qwen35_once_slot_aware(",
            "super::engine_qwen35::generate_stream_qwen35_once_extended_slot_aware(",
            "super::engine_qwen35::embed_qwen35_slot_aware(",
            "super::engine_qwen35::generate_qwen35_once_with_soft_tokens_slot_aware(",
        ] {
            assert!(
                body.contains(lift_fn),
                "H82 FALSIFIED: Qwen35 lift fn `{lift_fn}` is NOT \
                 called from worker_run. iter-1 must NOT regress any \
                 Qwen35 worker-arm lift (§6.1.27/28/29/30)."
            );
        }

        // ADR-040 iter-C2e (2026-05-30 §6.1.52) — Qwen3VL clamp
        // SHIPPED post-Gemma 4 iter-1 (Gemma 4 iter-1 commit
        // predates C2e). Sibling discipline pin: Gemma 4 iter-1 must
        // not REMOVE the C2e Qwen3VL clamp.
        assert!(
            body.contains(
                "matches!(loaded, LoadedModel::Qwen3VlText(_)) && handle.slot_id != SlotId(0)"
            ),
            "H82 FALSIFIED (post-C2e revision per §6.1.52): Qwen3VL \
             clamp missing from worker_run. iter-C2e SHIPPED 2026-05-30 \
             flipping the Qwen3VL SlotAware spawn arm to `Ok(Engine)` AND \
             adding the four worker-arm clamps; Gemma 4 iter-1 must NOT \
             regress the C2e clamp."
        );

        // ADR-040 iter-B4c-kernel iter-5 (§6.1.37 — TERMINAL Gemma 4
        // worker-arm lift) REVISES H82: post-iter-5 ALL FOUR Gemma 4
        // worker arms are lifted (Generate iter-1+2B + GenerateStream
        // iter-3 + Embed iter-4 + SoftTokens iter-5).  The SoftTokens
        // clamp label `gemma4-forward-prefill-with-soft-tokens-slot-N`
        // is LEGITIMATELY REMOVED by iter-5.  Sibling-discipline intent
        // ("iter-1 must not regress prior iters' lifts AND vice-versa")
        // preserved by pinning ALL iter-{1,3,4,5} Gemma 4 lift fns
        // are called from worker_run (positive assertions below);
        // mirror of iter-4 §6.1.36's H108 revision pattern that
        // converted the Embed-clamp-persisted assertion into the
        // Embed-lift-fn-present assertion when iter-4 lifted the Embed
        // arm.
        assert!(
            body.contains("generate_gemma4_once_with_soft_tokens_slot_aware("),
            "H82 FALSIFIED (post-iter-5 revision per §6.1.37): \
             Gemma 4 iter-5 SoftTokens lift fn \
             `generate_gemma4_once_with_soft_tokens_slot_aware` is \
             NOT called from worker_run. iter-1 must NOT regress \
             iter-5's TERMINAL SoftTokens-arm lift (§6.1.37)."
        );
        // iter-3 lift fn is called (B4c iter-3 §6.1.35 lifted the
        // Gemma 4 GenerateStream arm — pin the lift fn presence).
        assert!(
            body.contains("generate_stream_gemma4_once_slot_aware("),
            "H82 FALSIFIED: Gemma 4 iter-3 lift fn \
             `generate_stream_gemma4_once_slot_aware` is NOT called \
             from worker_run. iter-3 §6.1.35 lift was reverted — \
             GenerateStream slot-aware port is staged as \
             iter-B4c-kernel-iter-3 and MUST be wired post-iter-3."
        );
        // iter-4 lift fn is called (B4c iter-4 §6.1.36 lifted the
        // Gemma 4 Embed arm — pin the lift fn presence).
        assert!(
            body.contains("embed_gemma4_slot_aware("),
            "H82 FALSIFIED: Gemma 4 iter-4 lift fn \
             `embed_gemma4_slot_aware` is NOT called from worker_run. \
             iter-4 §6.1.36 lift was reverted — Embed slot-aware port \
             is staged as iter-B4c-kernel-iter-4 and MUST be wired \
             post-iter-4."
        );
    }

    /// **H83 (skip-mode)** — iter-1 sub-deferrals are NAMED for every
    /// remaining sub-iter:
    /// - `iter-B4c-kernel-iter-2`: the kernel-forward step itself,
    ///   typed-deferred inside the orchestrator body.
    /// - `iter-B4c-kernel-iter-3`: GenerateStream slot-aware port.
    /// - `iter-B4c-kernel-iter-4`: Embed slot-aware port.
    /// - `iter-B4c-kernel-iter-5`: GenerateWithSoftTokens slot-aware port.
    ///
    /// Drift here means a sub-deferral lost its operator-grep'able
    /// label.  ALSO pins that the §6.1.31 closure block exists in the
    /// ADR (forward pin — the §6.1.31 ADR block IS the destination
    /// of every `iter-B4c-kernel-iter-N per ADR-040 §6.1.31` cite).
    ///
    /// Mirrors iter-C2d-cont-kernel iter-1 H57.
    #[test]
    fn h83_iter1_sub_deferrals_named_for_remaining_iters() {
        let src = include_str!("engine.rs");
        // Every sub-iter cite must appear in worker_run (or the
        // orchestrator body).
        for label in [
            "iter-B4c-kernel-iter-2 per ADR-040 §6.1.31", // kernel-forward
            "iter-B4c-kernel-iter-3 per ADR-040 §6.1.31", // GenerateStream
            "iter-B4c-kernel-iter-4 per ADR-040 §6.1.31", // Embed
            "iter-B4c-kernel-iter-5 per ADR-040 §6.1.31", // SoftTokens
        ] {
            assert!(
                src.contains(label),
                "H83 FALSIFIED: sub-deferral label `{label}` is \
                 NOT present in engine.rs. iter-1's typed-deferral \
                 discipline broken — operator log greps cannot land \
                 on the right pin pointer."
            );
        }

        // §6.1.31 closure block exists in the ADR + names iter-B4c-
        // kernel iter-1 + the 4 sub-deferrals.
        let adr = crate::serve::api::engine::adr040_history_doc() /* iter-230 A1: §6.1.x moved to history (aeb6e87c) */;
        assert!(
            adr.contains("### 6.1.31"),
            "H83 FALSIFIED: ADR-040 §6.1.31 closure block not found. \
             Sub-deferral cites point at a non-existent destination."
        );
        let closure_marker = "### 6.1.31";
        let closure_start = adr
            .find(closure_marker)
            .expect("H83: §6.1.31 marker missing (asserted above)");
        let closure_end_off = adr[closure_start..]
            .find("\n### ")
            .unwrap_or_else(|| adr[closure_start..].len().min(40_000));
        let closure_body = &adr[closure_start..closure_start + closure_end_off];
        for required in [
            "iter-B4c-kernel iter-1",
            "iter-B4c-kernel-iter-2",
            "iter-B4c-kernel-iter-3",
            "iter-B4c-kernel-iter-4",
            "iter-B4c-kernel-iter-5",
        ] {
            assert!(
                closure_body.contains(required),
                "H83 FALSIFIED: ADR-040 §6.1.31 closure body does NOT \
                 name `{required}`. Sub-deferral runbook incomplete."
            );
        }
    }
}

// ============================================================================
// ADR-040 iter-B4c-kernel iter-2A — H84-H90 hypothesis pins
// ============================================================================
//
// Scope (iter-2A advances iter-1 by ONE call-graph hop):
//
//   * iter-1 (§6.1.31, commit `bac4c385`) shipped:
//     - `MultiSeqHbKvBuffers::reset_for_slot(slot)` + sibling
//       `MultiSeqHybridKvBuffers::reset_for_slot(slot)` primitives.
//     - `generate_gemma4_once_slot_aware` orchestrator scaffold —
//       bounds-checks slot_id + entry+exit reset_for_slot + typed-
//       deferred kernel-forward step as `iter-B4c-kernel-iter-2`.
//     - `worker_run` Gemma 4 Generate-arm lift fork (take + restore
//       borrow on `g.multi_seq_kv`).
//
//   * iter-2A (THIS commit, ADR-040 §6.1.32) ships:
//     - NEW `MlxModelWeights::forward_prefill_with_soft_tokens_slot_aware`
//       on `src/serve/forward_prefill.rs` — the load-bearing primitive
//       the iter-1 orchestrator's IIFE-wrapped typed-deferral now CALLS
//       (instead of surfacing the typed error at the orchestrator
//       boundary).
//     - Bounds-first pre-flight in the new fn (slot_id < n_seqs;
//       multi_seq_kv_hb.len() == self.layers.len(); empty prompt
//       guard).
//     - Dispatch fork on the 4 production KV regimes (hybrid F16-K +
//       TQ-HB-V; HB-encoded; legacy 4-bit; dense F32) — each branch
//       surfaces its own typed `MultiSeqError::CapabilityUnsupported`
//       with the named sub-iter (iter-B4c-kernel-iter-{2A-cont,2B,2C,2D}).
//     - Orchestrator update: replaces the iter-1 IIFE-wrapped typed
//       error with a real call into the new fn, propagating its typed
//       errors verbatim + naming the iter-2-decode sub-deferral on the
//       hypothetical Ok branch (never reached in iter-2A).
//
// Tests (H84-H90):
//   H84 (skip-mode): NEW fn `forward_prefill_with_soft_tokens_slot_aware`
//                    is DEFINED on MlxModelWeights with the correct
//                    signature (slot_id + multi_seq_kv_hb params).
//   H85 (skip-mode): Orchestrator `generate_gemma4_once_slot_aware`
//                    CALLS the new fn (one call-graph hop advance vs
//                    iter-1's IIFE typed error at orchestrator
//                    boundary).
//   H86 (skip-mode): SerialFifo + SlotId(0) byte-equivalence preserved
//                    — `forward_prefill_with_soft_tokens_resume` is
//                    NEVER called with `slot_id` (signature unchanged;
//                    existing call sites untouched).  Defends H1/H2/
//                    H23/H41/H44 byte-equivalence chain.
//   H87 (skip-mode): iter-2A typed sub-deferrals all NAMED (2A-cont,
//                    2B, 2C, 2D, 2-decode).
//   H88 (skip-mode): Bounds-first pre-flight (slot_id.0 < n_seqs) lands
//                    in the new fn body — A2b §6.1.23 iter-1.5
//                    cfa-finding-F5 ordering preserved.
//   H89 (skip-mode): Layer-count match pre-flight
//                    (multi_seq_kv_hb.len() == self.layers.len()) lands.
//   H90 (skip-mode): Orchestrator's iter-1 IIFE typed error
//                    `"gemma4-forward-prefill-kernel-slot-N
//                    (iter-B4c-kernel-iter-2 ..."` is REPLACED with the
//                    new fn call.  iter-1's typed error label REMOVED
//                    from the orchestrator body (would be a structural
//                    regression — the iter-1 deferral has been resolved).
// ----------------------------------------------------------------------------

#[cfg(test)]
mod adr040_phase_b_iter_b4c_kernel_iter2a_gemma4_tests {
    // Skip-mode source-grep tests; intentionally NO `use super::*;`
    // (tests rely on `include_str!` against engine.rs + forward_prefill.rs
    // + the ADR doc).

    /// **H84 (skip-mode)** — NEW
    /// `forward_prefill_with_soft_tokens_slot_aware` fn IS defined on
    /// `MlxModelWeights` in `src/serve/forward_prefill.rs` with the
    /// correct signature: `slot_id: SlotId` + `multi_seq_kv_hb: &mut
    /// Vec<MultiSeqHbKvBuffers>` parameters.
    ///
    /// Mirrors iter-1 H78's lift-witness shape for the model-fn level.
    #[test]
    fn h84_new_slot_aware_prefill_fn_landed_on_mlx_model_weights() {
        let src = include_str!("../forward_prefill.rs");
        assert!(
            src.contains("pub fn forward_prefill_with_soft_tokens_slot_aware("),
            "H84 FALSIFIED: `forward_prefill_with_soft_tokens_slot_aware` \
             is NOT defined as a pub fn in forward_prefill.rs. iter-2A \
             load-bearing primitive missing."
        );
        // Signature shape: takes `slot_id: SlotId` AND
        // `multi_seq_kv_hb: &mut Vec<MultiSeqHbKvBuffers>`.
        let fn_marker = "pub fn forward_prefill_with_soft_tokens_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H84: fn marker present (asserted above)");
        // ADR-040 iter-B4c-kernel iter-2C + iter-2D (§6.1.46) — sig
        // window bumped from 2_000 to 4_000 to accommodate the 2 new
        // Option<&mut Vec<MultiSeq{Dense,Mlx}KvBuffers>> params with
        // their docstrings.
        let sig_window = &src[fn_idx..(fn_idx + 4_000).min(src.len())];
        assert!(
            sig_window.contains("slot_id: SlotId"),
            "H84 FALSIFIED: new fn signature missing `slot_id: SlotId` \
             parameter. Per-slot routing is broken — the fn cannot \
             receive the admit'd handle's SlotId."
        );
        assert!(
            sig_window.contains("multi_seq_kv_hb: &mut Vec<MultiSeqHbKvBuffers>"),
            "H84 FALSIFIED: new fn signature missing `multi_seq_kv_hb: \
             &mut Vec<MultiSeqHbKvBuffers>` parameter. The persistent \
             multi-seq scaffold C2c §6.1.21 provisioned cannot be \
             consumed without this — iter-2A-cont kernel-dispatch \
             refactor has no destination."
        );
        // Returns Result<u32> (first decode token) — same shape as
        // the sibling forward_prefill_with_soft_tokens_resume.
        assert!(
            sig_window.contains(") -> Result<u32>"),
            "H84 FALSIFIED: new fn return type is not `Result<u32>`. \
             Sibling discipline broken — first-decode-token shape must \
             match `forward_prefill_with_soft_tokens_resume` so the \
             orchestrator decode-loop body (iter-2-decode) can wire \
             through verbatim."
        );
    }

    /// **H85 (skip-mode)** — Orchestrator
    /// `generate_gemma4_once_slot_aware` CALLS the new fn.  One
    /// call-graph hop advance vs iter-1: iter-1 IIFE-wrapped a typed
    /// `CapabilityUnsupported` at the orchestrator boundary; iter-2A
    /// replaces that with a real call into the model fn, which itself
    /// produces the typed deferral at the per-regime dispatch fork.
    ///
    /// Pin defends the regression where iter-2A accidentally
    /// regresses to iter-1 behaviour (typed error at orchestrator
    /// boundary instead of inside the new fn).
    #[test]
    fn h85_orchestrator_calls_new_slot_aware_prefill_fn() {
        let src = include_str!("engine.rs");
        let fn_marker = "fn generate_gemma4_once_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H85: generate_gemma4_once_slot_aware not found");
        let fn_window = &src[fn_idx..(fn_idx + 10_000).min(src.len())];
        // The new fn call site MUST be present.
        assert!(
            fn_window.contains(".forward_prefill_with_soft_tokens_slot_aware("),
            "H85 FALSIFIED: orchestrator does NOT call \
             `forward_prefill_with_soft_tokens_slot_aware`. iter-2A's \
             one-hop call-graph advance is not landed — orchestrator \
             would surface CapabilityUnsupported at its boundary \
             (iter-1 behaviour)."
        );
        // The call site MUST pass `slot_id` (the orchestrator's param)
        // — not a hard-coded SlotId(0) literal.
        let call_marker = ".forward_prefill_with_soft_tokens_slot_aware(";
        let call_idx = fn_window
            .find(call_marker)
            .expect("H85: call_marker present (asserted above)");
        let call_window = &fn_window[call_idx..(call_idx + 800).min(fn_window.len())];
        assert!(
            call_window.contains("slot_id"),
            "H85 FALSIFIED: orchestrator call site does NOT pass \
             `slot_id`. Per-slot routing broken — the new fn must \
             receive the orchestrator's SlotId."
        );
        // Hardness: the orchestrator must NOT pass a literal SlotId(0)
        // — that would silently route every slot through slot 0's
        // region of the multi-seq scaffold.
        assert!(
            !call_window.contains(", SlotId(0),"),
            "H85 FALSIFIED: orchestrator call site contains a literal \
             `SlotId(0)` argument. Per-slot routing is broken."
        );
        // The call site must pass `multi_seq_kv` (the orchestrator's
        // &mut Vec<MultiSeqHbKvBuffers> param).
        assert!(
            call_window.contains("multi_seq_kv"),
            "H85 FALSIFIED: orchestrator call site does NOT pass \
             `multi_seq_kv`. The persistent multi-seq scaffold cannot \
             be consumed by the new fn — iter-2A-cont kernel-dispatch \
             refactor has nothing to slice into."
        );
    }

    /// **H86 (skip-mode)** — SerialFifo + SlotId(0) byte-equivalence
    /// preserved.  The sibling fn
    /// `forward_prefill_with_soft_tokens_resume` is NEVER called with
    /// `slot_id` (its signature MUST remain unchanged; existing call
    /// sites at engine.rs:6463 + 7043 + 9769 untouched).
    ///
    /// Pin defends H1/H2/H23/H41/H44 byte-equivalence chain at the
    /// model-fn signature level: any modification to the sibling fn's
    /// signature would force every caller to be re-audited for byte-
    /// equivalence regression — instead, iter-2A enforces code-path
    /// disjointness via a NEW sibling.
    #[test]
    fn h86_serial_fifo_sibling_fn_signature_unchanged() {
        let src = include_str!("../forward_prefill.rs");
        // The sibling fn `forward_prefill_with_soft_tokens_resume` has
        // exactly its pre-iter-2A signature (5 args: prompt_tokens,
        // soft_tokens, max_decode_tokens, gpu, restored_lcp).
        let sibling_marker = "pub fn forward_prefill_with_soft_tokens_resume(";
        let sib_idx = src
            .find(sibling_marker)
            .expect("H86: sibling fn signature missing");
        let sib_window = &src[sib_idx..(sib_idx + 1000).min(src.len())];
        assert!(
            !sib_window.contains("slot_id"),
            "H86 FALSIFIED: `forward_prefill_with_soft_tokens_resume` \
             signature contains `slot_id` parameter. iter-2A discipline \
             broken — the sibling fn MUST remain byte-equivalent for \
             SerialFifo + SlotId(0).  iter-2A's primitive is a NEW \
             sibling fn (`forward_prefill_with_soft_tokens_slot_aware`); \
             the existing sibling MUST NOT be touched."
        );
        assert!(
            !sib_window.contains("multi_seq_kv"),
            "H86 FALSIFIED: `forward_prefill_with_soft_tokens_resume` \
             signature mentions `multi_seq_kv`. iter-2A discipline \
             broken — SerialFifo path MUST NOT consume the multi-seq \
             scaffold."
        );
        // The 3 production call sites in engine.rs MUST still call
        // `forward_prefill_with_soft_tokens_resume` (NOT the new slot-
        // aware variant), preserving byte-equivalence for the
        // non-slot-aware paths.
        let engine_src = include_str!("engine.rs");
        let n_resume_calls = engine_src
            .matches(".forward_prefill_with_soft_tokens_resume(")
            .count();
        assert!(
            n_resume_calls >= 2,
            "H86 FALSIFIED: pre-iter-2A engine.rs had ≥2 call sites \
             of `forward_prefill_with_soft_tokens_resume` (at \
             generate_once + LCP fast paths). Post-iter-2A count is \
             {n_resume_calls} — call sites silently rerouted, byte- \
             equivalence chain compromised."
        );
    }

    /// **H87 (skip-mode; REVISED 2026-05-30 §6.1.38)** — iter-2A typed
    /// sub-deferrals are all NAMED with operator-grep'able iter-N labels
    /// for each remaining sub-iter.  Mirror of iter-1 H83's discipline.
    ///
    /// **REVISION POST-iter-2-decode-A (§6.1.38)**: the literal
    /// `iter-B4c-kernel-iter-2-decode per ADR-040 §6.1.32` was REMOVED
    /// from the orchestrator IIFE — iter-2-decode-A landed real decode-
    /// loop bodies in all 3 orchestrators (Generate / GenerateStream /
    /// SoftTokens) replacing the iter-2-decode IIFE typed-error returns.
    /// The orchestrator decode-loop body now carries an
    /// `iter-B4c-kernel-iter-2-decode-C per ADR-040 §6.1.38` cite (the
    /// surviving sub-deferral: full sampler/grammar/tool-call/stop-strings
    /// /logprobs/reasoning-text surface).  Sibling-discipline intent
    /// preserved by swapping the literal — H87 still pins that EVERY
    /// 4-way dispatch-fork branch on the prefill side AND a surviving
    /// decode-side sub-deferral are NAMED.
    ///
    /// - `iter-B4c-kernel-iter-2A-cont`: HB-encoded prefill slot
    ///   routing (the in-scope kernel-dispatch refactor surface).
    /// - `iter-B4c-kernel-iter-2B`: HybridKvBuffers slot routing
    ///   (HF2Q_HYBRID_KV=1 production-default per H10 falsification).
    /// - `iter-B4c-kernel-iter-2C`: legacy 4-bit path
    ///   (HF2Q_TQ_CODEBOOK_BITS=4 opt-in surface).
    /// - `iter-B4c-kernel-iter-2D`: dense F32 path (HF2Q_USE_DENSE=1
    ///   LCP-eligible regime).
    /// - `iter-B4c-kernel-iter-2-decode-C-stream-tool-call`: streaming
    ///   tool-call body emission surface via ToolCallStreamEmitter
    ///   (REVISED at iter-2-decode-C SHIP per §6.1.39 — iter-2-decode-A's
    ///   `iter-2-decode-C` orchestrator-wide sampling/grammar/logprobs/
    ///   reasoning-text label LIFTED into the production-engagement
    ///   greedy + sampled non-streaming + sampled streaming surface;
    ///   only the streaming tool-call body emission via the Wave 3 W-B3
    ///   `ToolCallStreamEmitter` remains a typed sub-deferral).
    ///
    /// Drift here means a sub-deferral lost its operator-grep'able
    /// label.
    #[test]
    fn h87_iter2a_sub_deferrals_named_for_remaining_iters() {
        let pf_src = include_str!("../forward_prefill.rs");
        let engine_src = include_str!("engine.rs");
        let combined = format!("{pf_src}\n{engine_src}");
        for label in [
            // 4 dispatch-fork branches inside the new fn body.
            "iter-B4c-kernel-iter-2A-cont per ADR-040 §6.1.32", // HB-encoded
            "iter-B4c-kernel-iter-2B per ADR-040 §6.1.32",      // HybridKvBuffers
            "iter-B4c-kernel-iter-2C per ADR-040 §6.1.32",      // legacy 4-bit
            "iter-B4c-kernel-iter-2D per ADR-040 §6.1.32",      // dense F32
            // Surviving orchestrator sub-deferral — REVISED at iter-
            // 2-decode-C SHIP: iter-2-decode-A's `iter-2-decode-C`
            // orchestrator-wide label was REPLACED with the real
            // sampler/grammar/stop-strings/logprobs/reasoning-text
            // surface; only the streaming tool-call body emission
            // via Wave 3 W-B3's `ToolCallStreamEmitter` remains a
            // typed sub-deferral (~200 LOC of stateful incremental
            // JSON parsing; deferred to keep iter-2-decode-C
            // structurally bounded).
            "iter-B4c-kernel-iter-2-decode-C-stream-tool-call per ADR-040 §6.1.39",
        ] {
            assert!(
                combined.contains(label),
                "H87 FALSIFIED: sub-deferral label `{label}` is NOT \
                 present in forward_prefill.rs or engine.rs. iter-2A's \
                 typed-deferral discipline broken — operator log greps \
                 cannot land on the right pin pointer."
            );
        }
    }

    /// **H88 (skip-mode)** — Bounds-first pre-flight per A2b §6.1.23
    /// iter-1.5 cfa-finding-F5 ordering preserved.  The new fn checks
    /// `slot_id.0 < multi_seq_kv_hb[0].n_seqs` (or equivalent) BEFORE
    /// any other body-level work begins — mirrors the Qwen35 B4a
    /// contract at `forward_gpu.rs:2569-2586`.
    ///
    /// Pin defends silent-corruption regressions where a stale slot_id
    /// (from a stale handle) would index past the scaffold's n_seqs
    /// and silently corrupt slot N's K/V region.
    #[test]
    fn h88_new_fn_bounds_first_preflight_lands() {
        let src = include_str!("../forward_prefill.rs");
        let fn_marker = "pub fn forward_prefill_with_soft_tokens_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H88: fn marker present (H84 asserts)");
        // ADR-040 iter-2C + iter-2D (§6.1.46) — window bumped from
        // 12_000 to 50_000 to cover the bounds preflight AND the
        // INVESTIGATION_ENV.hybrid_kv dispatch fork, which now sits
        // after the new dense F32 + legacy 4-bit branches (the hybrid
        // branch is now at line ~3187, ~37K bytes past fn start).
        let body_window = &src[fn_idx..(fn_idx + 50_000).min(src.len())];
        // The bounds check reads `multi_seq_kv_hb[0].n_seqs` and
        // compares against `slot_id.0`.
        assert!(
            body_window.contains("n_seqs = multi_seq_kv_hb[0].n_seqs"),
            "H88 FALSIFIED: new fn does not bind n_seqs from \
             `multi_seq_kv_hb[0].n_seqs`. Bounds-first preflight cannot \
             use the canonical n_seqs source."
        );
        assert!(
            body_window.contains("slot_id.0 >= n_seqs"),
            "H88 FALSIFIED: new fn does not check `slot_id.0 >= n_seqs` \
             — bounds-first preflight is broken. A stale slot_id could \
             silently corrupt the wrong slot's K/V region."
        );
        // Per A2b iter-1.5 ordering: bounds check fires BEFORE any
        // kernel-dispatch work.  We approximate via lexical ordering:
        // the bounds check string appears BEFORE the dispatch fork
        // (the `INVESTIGATION_ENV.hybrid_kv` branch).
        let bounds_pos = body_window
            .find("slot_id.0 >= n_seqs")
            .expect("H88: bounds check present (asserted above)");
        let dispatch_pos = body_window
            .find("INVESTIGATION_ENV.hybrid_kv")
            .expect("H88: dispatch fork present (must be lexically AFTER bounds check)");
        assert!(
            bounds_pos < dispatch_pos,
            "H88 FALSIFIED: bounds-first ordering violated — the \
             dispatch fork at `INVESTIGATION_ENV.hybrid_kv` appears \
             BEFORE the bounds check at `slot_id.0 >= n_seqs`. A2b \
             §6.1.23 iter-1.5 cfa-finding-F5 ordering broken."
        );
    }

    /// **H89 (skip-mode)** — Layer-count match pre-flight lands.  The
    /// new fn asserts `multi_seq_kv_hb.len() == self.layers.len()` as
    /// the caller-invariant defense-in-depth check.  C2c spawn-arm
    /// produces exactly one entry per layer per the provisioning loop
    /// at `engine.rs::provision_multi_seq_kv_for_slot_aware`; a desync
    /// would silently route the scaffold's per-layer K/V buffers to
    /// the wrong layer.
    #[test]
    fn h89_new_fn_layer_count_match_preflight_lands() {
        let src = include_str!("../forward_prefill.rs");
        let fn_marker = "pub fn forward_prefill_with_soft_tokens_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H89: fn marker present (H84 asserts)");
        let body_window = &src[fn_idx..(fn_idx + 12_000).min(src.len())];
        assert!(
            body_window.contains("multi_seq_kv_hb.len() != self.layers.len()"),
            "H89 FALSIFIED: new fn does not check \
             `multi_seq_kv_hb.len() != self.layers.len()`. A C2c \
             spawn-arm desync (e.g. layer-count mismatch from a partial \
             provisioning) would silently route layer-N's K/V to \
             layer-M's buffer."
        );
        // ALSO pins the empty-scaffold defense (defense-in-depth — the
        // orchestrator at §6.1.31 also checks this; the new fn re-
        // checks so a future iter-2A-cont edit that lifts the
        // orchestrator's check doesn't accidentally remove BOTH
        // surfaces).
        assert!(
            body_window.contains("multi_seq_kv_hb.is_empty()"),
            "H89 FALSIFIED: new fn does not check \
             `multi_seq_kv_hb.is_empty()` defense-in-depth."
        );
    }

    /// **H90 (skip-mode)** — Orchestrator's iter-1 IIFE typed error
    /// `"gemma4-forward-prefill-kernel-slot-N (iter-B4c-kernel-iter-2 ..."`
    /// is REPLACED with the new fn call.  iter-1's typed-error label
    /// no longer appears at the orchestrator boundary — the typed
    /// error now surfaces from INSIDE the new fn (at the per-regime
    /// dispatch fork), one call-graph hop further down.
    ///
    /// This is the load-bearing structural-advance pin: iter-2A's
    /// scope is precisely "advance the typed-deferral by one
    /// call-graph hop"; H90 checks the advance landed.
    #[test]
    fn h90_orchestrator_iter1_typed_error_replaced_with_new_fn_call() {
        let src = include_str!("engine.rs");
        let fn_marker = "fn generate_gemma4_once_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H90: generate_gemma4_once_slot_aware not found");
        let fn_window = &src[fn_idx..(fn_idx + 10_000).min(src.len())];
        // The iter-1 IIFE-wrapped typed-error label is the literal
        // `"gemma4-forward-prefill-kernel-slot-N (iter-B4c-kernel-iter-2 "`
        // (note the trailing space distinguishes it from
        // `iter-B4c-kernel-iter-2A-cont` / `iter-2-decode` / etc).
        let iter1_label = "gemma4-forward-prefill-kernel-slot-N (iter-B4c-kernel-iter-2 per";
        assert!(
            !fn_window.contains(iter1_label),
            "H90 FALSIFIED: orchestrator body still contains iter-1's \
             typed-error label `{iter1_label}`. iter-2A's call-graph \
             advance is NOT landed — typed error still surfaces at \
             orchestrator boundary instead of from inside the new \
             model fn."
        );
        // Positive pin: the new fn call IS present in the orchestrator
        // body (H85 also checks this; H90 re-asserts to bind the
        // two-part discipline: REMOVE iter-1 label AND ADD new fn call).
        assert!(
            fn_window.contains(".forward_prefill_with_soft_tokens_slot_aware("),
            "H90 FALSIFIED: orchestrator does NOT call the new fn. \
             Structural advance broken — orchestrator still in iter-1 \
             behaviour."
        );
        // ALSO pins that the §6.1.32 ADR block exists.
        let adr = crate::serve::api::engine::adr040_history_doc() /* iter-230 A1: §6.1.x moved to history (aeb6e87c) */;
        assert!(
            adr.contains("### 6.1.32"),
            "H90 FALSIFIED: ADR-040 §6.1.32 closure block not found. \
             iter-2A's sub-deferral cites point at a non-existent \
             destination."
        );
        let closure_marker = "### 6.1.32";
        let closure_start = adr
            .find(closure_marker)
            .expect("H90: §6.1.32 marker missing (asserted above)");
        let closure_end_off = adr[closure_start..]
            .find("\n### ")
            .unwrap_or_else(|| adr[closure_start..].len().min(40_000));
        let closure_body = &adr[closure_start..closure_start + closure_end_off];
        for required in [
            "iter-B4c-kernel iter-2A",
            "iter-B4c-kernel-iter-2A-cont",
            "iter-B4c-kernel-iter-2B",
            "iter-B4c-kernel-iter-2C",
            "iter-B4c-kernel-iter-2D",
            "iter-B4c-kernel-iter-2-decode",
        ] {
            assert!(
                closure_body.contains(required),
                "H90 FALSIFIED: ADR-040 §6.1.32 closure body does NOT \
                 name `{required}`. Sub-deferral runbook incomplete."
            );
        }
    }
}

// ============================================================================
// ADR-040 iter-C2c-cont — H91-H96 hypothesis pins
// ============================================================================
//
// Scope (iter-C2c-cont extends C2c §6.1.21's `GemmaLoadedModel.multi_seq_kv`
// HbKvBuffers scaffold with a SIBLING `multi_seq_kv_hybrid` field carrying
// the production-default MultiSeqHybridKvBuffers scaffold per H10
// falsification at §6.1.11 — `HF2Q_HYBRID_KV` is default-true since ADR-029
// iter-13, 2026-05-11).
//
//   * C2c (§6.1.21, commit `a0540b28`) shipped:
//     - `GemmaLoadedModel.multi_seq_kv: Option<Vec<MultiSeqHbKvBuffers>>`
//       provisioned at spawn time via `provision_multi_seq_kv_for_slot_aware`
//       through the A3a `alloc_hb_kv_for_layer` allocator.
//     - `EngineSpawnError::Gemma4SlotAwareProvisionFailed { max_slots, cause }`
//       typed variant on allocator failure.
//     - SerialFifo path leaves `multi_seq_kv = None` (H23 byte-equivalence pin).
//
//   * iter-B4c-kernel iter-2A (§6.1.32, commit `6a5b7ca4`) surfaced the gap:
//     - The new `forward_prefill_with_soft_tokens_slot_aware` fn body reads
//       `INVESTIGATION_ENV.hybrid_kv` and would route the production-default
//       request through the `MultiSeqHybridKvBuffers` regime — but there's
//       NO field on `GemmaLoadedModel` carrying that scaffold (C2c only
//       provisions HB).
//     - iter-2A's dispatch fork therefore surfaces typed CapabilityUnsupported
//       at the hybrid_kv branch with deferral label naming this iter
//       (iter-C2c-cont) as the upstream prerequisite.
//
//   * iter-C2c-cont (THIS commit, ADR-040 §6.1.33) ships:
//     - NEW `GemmaLoadedModel.multi_seq_kv_hybrid: Option<Vec<MultiSeqHybridKvBuffers>>`
//       sibling field (additive — C2c's `multi_seq_kv` field is PRESERVED
//       verbatim per H94).
//     - NEW `EngineSpawnError::Gemma4HybridSlotAwareProvisionFailed`
//       typed variant for the per-layer hybrid allocator's failure surface.
//     - Extended `provision_multi_seq_kv_for_slot_aware` body: Phase 1
//       provisions HB scaffold unconditionally (C2c preserved); Phase 2
//       provisions hybrid scaffold IFF `INVESTIGATION_ENV.hybrid_kv == true`
//       (PRODUCTION DEFAULT). The two phases reuse the SAME per-layer
//       `(nkv, hd, capacity, is_ring)` quadruples so a future iter-2B
//       kernel refactor inherits the iter-2A-cont addressing scheme.
//     - Extended spawn-arm body in `Engine::spawn_with_mode`: on
//       provisioning error, inspects whether `multi_seq_kv.is_some()` to
//       decide which typed-error variant to surface (HB Phase 1 vs hybrid
//       Phase 2). HB-phase failure preserves the pre-iter-C2c-cont contract
//       byte-for-byte (H22 / H29 string-format pins).
//
// Tests (H91-H96):
//   H91 (skip-mode): NEW field `multi_seq_kv_hybrid` IS DEFINED on
//                    `GemmaLoadedModel` with the correct type
//                    (`Option<Vec<MultiSeqHybridKvBuffers>>`); the C2c
//                    sibling `multi_seq_kv` is PRESERVED verbatim
//                    (H94 PRESERVED).
//   H92 (env-driven runtime): under `HF2Q_HYBRID_KV=1` (default) post-
//                    `provision_multi_seq_kv_for_slot_aware`, BOTH fields
//                    are `Some(_)`; under `HF2Q_HYBRID_KV=0`, only the
//                    HB sibling is populated. Exercised with a real Mlx
//                    device construction (no model load) so no OOM.
//   H93 (compile + structure): NEW typed-error variant
//                    `Gemma4HybridSlotAwareProvisionFailed { max_slots,
//                    cause }` is constructible at the type level; the
//                    Display contract names the iter-C2c-cont arc.
//   H94 (skip-mode): C2c HbKvBuffers provisioning surface PRESERVED —
//                    the C2c `multi_seq_kv` field is still present, its
//                    docstring's "iter-2c (C2c)" cite is still present,
//                    and `Gemma4SlotAwareProvisionFailed` variant is
//                    untouched. Defends against an iter-C2c-cont commit
//                    that accidentally renames/removes C2c's surface.
//   H95 (skip-mode): SerialFifo + Gemma 4 spawn does NOT provision the
//                    hybrid scaffold (sibling to H23: source-grep on
//                    `spawn_with_mode` SerialFifo arm asserts the
//                    `multi_seq_kv_hybrid` field is NOT touched there).
//   H96 (skip-mode): Qwen35 + Qwen3VL surfaces UNCHANGED — no
//                    `multi_seq_kv_hybrid` field on Qwen35LoadedModel
//                    / Qwen3VlText structs; no
//                    `Gemma4HybridSlotAwareProvisionFailed` reference
//                    in any Qwen35/Qwen3VL handler arm.
// ----------------------------------------------------------------------------

#[cfg(test)]
mod adr040_phase_c_iter_c2c_cont_gemma4_hybrid_provisioning_tests {
    // Skip-mode source-grep + structural tests; intentionally NO `use
    // super::*;` for the source-grep helpers, but a few synthesis-time
    // tests need the typed surface — those bring in `super::*` locally.

    /// **H91 (skip-mode)** — NEW field `multi_seq_kv_hybrid` IS DEFINED
    /// on `GemmaLoadedModel` with the correct type
    /// (`Option<Vec<MultiSeqHybridKvBuffers>>`).  The C2c sibling
    /// `multi_seq_kv: Option<Vec<MultiSeqHbKvBuffers>>` is PRESERVED
    /// verbatim (H94 PRESERVED — additive, NOT a replacement).
    ///
    /// Mirrors iter-2A H84's shape for the new struct field instead of
    /// a new fn signature.
    #[test]
    fn h91_new_multi_seq_kv_hybrid_field_defined_on_gemma_loaded_model() {
        let src = include_str!("engine.rs");
        let struct_marker = "pub struct GemmaLoadedModel {";
        let struct_idx = src
            .find(struct_marker)
            .expect("H91: GemmaLoadedModel struct not found in engine.rs");
        let struct_end = src[struct_idx..]
            .find("\n}\n")
            .expect("H91: GemmaLoadedModel struct close brace not found");
        let struct_window = &src[struct_idx..struct_idx + struct_end];
        let compact: String = struct_window
            .chars()
            .filter(|character| !character.is_whitespace())
            .collect();
        // C2c field PRESERVED.
        assert!(
            compact.contains(
                "pubmulti_seq_kv:Option<Vec<crate::inference::models::gemma4::kv_cache::MultiSeqHbKvBuffers>>"
            ),
            "H91 FALSIFIED: C2c `multi_seq_kv: Option<Vec<MultiSeqHbKvBuffers>>` \
             field is MISSING — H94 PRESERVED constraint violated. \
             iter-C2c-cont must be ADDITIVE."
        );
        // NEW iter-C2c-cont field PRESENT.
        assert!(
            compact.contains(
                "pubmulti_seq_kv_hybrid:Option<Vec<crate::inference::models::gemma4::kv_cache::MultiSeqHybridKvBuffers>>"
            ),
            "H91 FALSIFIED: NEW field `multi_seq_kv_hybrid: Option<Vec<\
             MultiSeqHybridKvBuffers>>` is MISSING from GemmaLoadedModel. \
             iter-C2c-cont load-bearing primitive not landed — iter-2B \
             hybrid kernel-dispatch refactor has no destination."
        );
    }

    /// **H92 (skip-mode + runtime when MLX available)** — provisioning
    /// honours `INVESTIGATION_ENV.hybrid_kv`:
    /// - Source-grep: the body of
    ///   `provision_multi_seq_kv_for_slot_aware` contains an
    ///   `INVESTIGATION_ENV.hybrid_kv` gate around the
    ///   `alloc_multi_seq_hybrid_kv_for_layer` call site.
    /// - Source-grep: the `multi_seq_kv` (HB) scaffold is provisioned
    ///   UNCONDITIONALLY (no env gate around `alloc_hb_kv_for_layer`).
    /// - Source-grep: the `multi_seq_kv_hybrid` field is assigned
    ///   `Some(_)` ONLY inside the env gate.
    ///
    /// Reasoning: the actual runtime semantic (Some/None per env) is
    /// what we want to pin, but we can't construct a real
    /// `GemmaLoadedModel` without loading a Gemma 4 GGUF (out of scope
    /// per the CLAUDE.md "do not oom us" rule). The source-grep pin
    /// catches the lexical structure that produces the runtime
    /// semantic.
    #[test]
    fn h92_hybrid_provisioning_gated_on_investigation_env_hybrid_kv() {
        let src = include_str!("engine.rs");
        let fn_marker = "pub fn provision_multi_seq_kv_for_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H92: provision_multi_seq_kv_for_slot_aware not found");
        // The fn body is large after iter-C2c-cont (~150 LOC); window
        // to 12k chars to cover docstring + body.
        let body_window = &src[fn_idx..(fn_idx + 12_000).min(src.len())];
        // The hybrid allocator IS called.
        assert!(
            body_window.contains("alloc_multi_seq_hybrid_kv_for_layer("),
            "H92 FALSIFIED: provision_multi_seq_kv_for_slot_aware body \
             does NOT call `alloc_multi_seq_hybrid_kv_for_layer`. \
             iter-C2c-cont production-default scaffold (per H10 \
             falsification §6.1.11) is NOT provisioned."
        );
        // The hybrid allocator call is gated on INVESTIGATION_ENV.hybrid_kv.
        let env_gate_marker = "INVESTIGATION_ENV.hybrid_kv";
        let env_gate_idx = body_window.find(env_gate_marker).expect(
            "H92 FALSIFIED: provision_multi_seq_kv_for_slot_aware body does \
             NOT contain `INVESTIGATION_ENV.hybrid_kv` — the hybrid \
             provisioning is unconditional (BREAKING H95: HF2Q_HYBRID_KV=0 \
             must not allocate the hybrid scaffold).",
        );
        let alloc_idx = body_window
            .find("alloc_multi_seq_hybrid_kv_for_layer(")
            .expect("H92: alloc call present (asserted above)");
        assert!(
            env_gate_idx < alloc_idx,
            "H92 FALSIFIED: `INVESTIGATION_ENV.hybrid_kv` does NOT \
             lexically precede `alloc_multi_seq_hybrid_kv_for_layer` \
             call. The env gate must wrap the alloc — without lexical \
             ordering, the alloc is not gated and HF2Q_HYBRID_KV=0 \
             would still allocate hybrid bytes."
        );
        // The HB allocator IS called UNCONDITIONALLY (not inside the
        // env gate). Source-order check: HB alloc comes BEFORE the env
        // gate in the fn body (Phase 1 always; Phase 2 conditional).
        let hb_alloc_idx = body_window
            .find("alloc_hb_kv_for_layer(")
            .expect("H92: alloc_hb_kv_for_layer call present in fn body");
        assert!(
            hb_alloc_idx < env_gate_idx,
            "H92 FALSIFIED: `alloc_hb_kv_for_layer` does NOT lexically \
             precede `INVESTIGATION_ENV.hybrid_kv` env gate. C2c HB \
             provisioning must be UNCONDITIONAL (H94 preserved); \
             accidentally moving it inside the env gate would break \
             the HF2Q_HYBRID_KV=0 path."
        );
        // The `multi_seq_kv_hybrid = Some(_)` assignment is inside the
        // env-gated branch (positive: assignment present at all).
        assert!(
            body_window.contains("self.multi_seq_kv_hybrid = Some("),
            "H92 FALSIFIED: provision_multi_seq_kv_for_slot_aware body \
             does NOT assign `self.multi_seq_kv_hybrid = Some(_)`. The \
             new field is never populated — iter-C2c-cont effectively \
             unimplemented."
        );
        // Defense-in-depth: the C2c HB assignment is unchanged.
        assert!(
            body_window.contains("self.multi_seq_kv = Some("),
            "H92 FALSIFIED: C2c assignment `self.multi_seq_kv = Some(_)` \
             removed — H94 broken."
        );
    }

    /// **H93 (compile pin + Display contract)** — NEW typed-error
    /// variant `Gemma4HybridSlotAwareProvisionFailed { max_slots: u32,
    /// cause: String }` is CONSTRUCTIBLE at the type level + carries
    /// both fields + its Display message names the iter-C2c-cont arc +
    /// the production-default hybrid F16-K + TQ-HB-V regime.
    #[test]
    fn h93_gemma4_hybrid_provision_failed_variant_carries_max_slots_and_cause() {
        use super::EngineSpawnError;
        let err = EngineSpawnError::Gemma4HybridSlotAwareProvisionFailed {
            max_slots: 4,
            cause: "synthetic-cause: alloc_multi_seq_hybrid_kv_for_layer L0 OOM".to_string(),
        };
        // Variant destructures with the expected field shape.
        match &err {
            EngineSpawnError::Gemma4HybridSlotAwareProvisionFailed { max_slots, cause } => {
                assert_eq!(*max_slots, 4u32, "H93 sanity: max_slots round-trips");
                assert!(cause.contains("OOM"), "H93 sanity: cause round-trips");
            }
            other => panic!(
                "H93 FALSIFIED: Gemma4HybridSlotAwareProvisionFailed \
                 variant does not destructure as expected; got {:?}",
                other
            ),
        }
        // Display message names the iter cite + the production-default
        // regime so operator log greps land on the right pin.
        let msg = format!("{}", err);
        for required in [
            "iter-C2c-cont",
            "Gemma 4",
            "MultiSeqHybridKvBuffers",
            "HF2Q_HYBRID_KV=1",
            "H10",
            "§6.1.11",
            "max_slots=4",
        ] {
            assert!(
                msg.contains(required),
                "H93 FALSIFIED: Display message does NOT contain `{required}`. \
                 Operator log greps cannot route to the right pin pointer. \
                 Got: {msg}"
            );
        }
        // Distinct from the C2c sibling variant — pin defends against
        // accidentally collapsing the two into one discriminant.
        let hb_err = EngineSpawnError::Gemma4SlotAwareProvisionFailed {
            max_slots: 4,
            cause: "hb cause".to_string(),
        };
        assert!(
            !format!("{}", hb_err).contains("iter-C2c-cont"),
            "H93 FALSIFIED: the C2c sibling variant's Display message \
             contains `iter-C2c-cont` — discriminant collapse risk. \
             Each variant must own its iter cite."
        );
    }

    /// **H94 (skip-mode)** — C2c HbKvBuffers provisioning surface is
    /// PRESERVED VERBATIM:
    /// - `Gemma4SlotAwareProvisionFailed` variant is still defined.
    /// - `multi_seq_kv: Option<Vec<MultiSeqHbKvBuffers>>` field is
    ///   still present (also asserted by H91).
    /// - `alloc_hb_kv_for_layer` is still called inside
    ///   `provision_multi_seq_kv_for_slot_aware`.
    /// - The "ADR-040 C2c:" diagnostic prefix on the HB error path is
    ///   preserved.
    ///
    /// Defends against an iter-C2c-cont commit that accidentally
    /// renames/removes any part of C2c's surface (the H22 / H23 / H25 /
    /// H29 chain depends on these strings).
    #[test]
    fn h94_c2c_hb_provisioning_surface_preserved() {
        let src = include_str!("engine.rs");
        let compact: String = src
            .chars()
            .filter(|character| !character.is_whitespace())
            .collect();
        // C2c typed-error variant present.
        assert!(
            src.contains("Gemma4SlotAwareProvisionFailed {"),
            "H94 FALSIFIED: C2c `Gemma4SlotAwareProvisionFailed` typed \
             variant removed — H21/H22/H29 break."
        );
        // C2c field present with correct element type.
        assert!(
            compact.contains(
                "pubmulti_seq_kv:Option<Vec<crate::inference::models::gemma4::kv_cache::MultiSeqHbKvBuffers>>"
            ),
            "H94 FALSIFIED: `multi_seq_kv` field declaration changed; \
             H22 access pattern broken."
        );
        // C2c HB allocator still called.
        assert!(
            src.contains("alloc_hb_kv_for_layer("),
            "H94 FALSIFIED: `alloc_hb_kv_for_layer` no longer called \
             from provision_multi_seq_kv_for_slot_aware — H22 \
             (`n_seqs == max_slots`) cannot be satisfied."
        );
        // C2c diagnostic context preserved.
        assert!(
            src.contains("ADR-040 C2c: alloc_hb_kv_for_layer L"),
            "H94 FALSIFIED: C2c HB error context message changed; \
             operator log greps for `ADR-040 C2c:` would miss."
        );
        // C2c HB assignment preserved (Phase 1 unconditional).
        assert!(
            src.contains("self.multi_seq_kv = Some(multi_seq);"),
            "H94 FALSIFIED: `self.multi_seq_kv = Some(multi_seq);` \
             assignment removed — C2c populates None."
        );
    }

    /// **H95 (skip-mode)** — SerialFifo Gemma 4 spawn does NOT touch
    /// the hybrid scaffold (sibling to H23 for the HB scaffold).
    /// Source-grep: the `spawn_with_mode` `SerialFifo` arm body does
    /// NOT call `provision_multi_seq_kv_for_slot_aware` AND does NOT
    /// reference `multi_seq_kv_hybrid`. SerialFifo byte-equivalence
    /// preserved at the spawn-arm level.
    ///
    /// Mirrors C2c H23 (the HB-side sibling pin) — A5d's source-order
    /// regression-pin pattern.
    #[test]
    fn h95_serial_fifo_does_not_provision_multi_seq_kv_hybrid() {
        let src = include_str!("engine.rs");
        // Find the spawn_with_mode fn body.
        let fn_marker = "pub fn spawn_with_mode(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H95: spawn_with_mode not found in engine.rs");
        // Window covers the full match block (~7000 chars).
        let fn_window = &src[fn_idx..(fn_idx + 12_000).min(src.len())];
        // Find the SerialFifo arm. Per the spawn_with_mode body, the
        // arm matches `EngineMode::SerialFifo` and delegates to
        // `Self::spawn(...)` (the legacy 3-arg constructor). The arm
        // body MUST NOT mention either provisioning fn or the new field.
        let fifo_arm_marker = "EngineMode::SerialFifo";
        let fifo_idx = fn_window
            .find(fifo_arm_marker)
            .expect("H95: SerialFifo arm not found in spawn_with_mode");
        let slot_aware_idx = fn_window[fifo_idx..]
            .find("EngineMode::SlotAware { max_slots }")
            .map(|i| fifo_idx + i)
            .unwrap_or(fn_window.len());
        // Window: SerialFifo arm body (everything between SerialFifo
        // and SlotAware match-arm markers).
        let fifo_arm = &fn_window[fifo_idx..slot_aware_idx];
        // Neither the provisioner nor the new field is referenced
        // inside the SerialFifo arm.
        assert!(
            !fifo_arm.contains("provision_multi_seq_kv_for_slot_aware"),
            "H95 FALSIFIED: SerialFifo arm of spawn_with_mode calls \
             `provision_multi_seq_kv_for_slot_aware` — SerialFifo \
             byte-equivalence broken (H23 sibling)."
        );
        assert!(
            !fifo_arm.contains("multi_seq_kv_hybrid"),
            "H95 FALSIFIED: SerialFifo arm of spawn_with_mode \
             references `multi_seq_kv_hybrid` — the hybrid scaffold \
             must remain `None` for SerialFifo (pre-ADR-040 byte-\
             equivalence)."
        );
        // ALSO pin the constructor: `GemmaLoadedModel::load` sets
        // multi_seq_kv_hybrid = None (sibling to multi_seq_kv = None).
        // SerialFifo's spawn path runs `load` (no per-arch dispatch),
        // never touches the field.
        let load_marker = "fn load(opts: &LoadOptions)";
        let load_idx = src
            .find(load_marker)
            .expect("H95: GemmaLoadedModel::load fn not found (engine.rs structure changed)");
        let load_window = &src[load_idx..(load_idx + 30_000).min(src.len())];
        assert!(
            load_window.contains("multi_seq_kv_hybrid: None,"),
            "H95 FALSIFIED: GemmaLoadedModel::load does NOT initialize \
             `multi_seq_kv_hybrid: None`. SerialFifo path enters the \
             worker thread with the field uninitialized (compile-fail) \
             OR worse, populated by a previous code path."
        );
    }

    /// **H96 (skip-mode)** — Qwen35 + Qwen3VL surfaces UNCHANGED by
    /// iter-C2c-cont:
    /// - No `multi_seq_kv_hybrid` field on `Qwen35LoadedModel` or
    ///   `Qwen3VlTextLoadedModel`.
    /// - No reference to `Gemma4HybridSlotAwareProvisionFailed` in
    ///   the Qwen35/Qwen3VL worker arms.
    /// - The `Qwen35SlotAwareProvisionFailed` variant string-format
    ///   contract preserved (H29 sibling).
    /// - No `alloc_multi_seq_hybrid_kv_for_layer` call inside the
    ///   `Qwen35LoadedModel::provision_multi_seq_kv_for_slot_aware`
    ///   implementation (the hybrid allocator is a Gemma 4 module
    ///   primitive).
    ///
    /// Defends sibling-discipline pin: iter-C2c-cont is a Gemma 4-only
    /// scaffold extension; Qwen35 has its own
    /// `HybridKvCache::new_with_options(.., n_seqs=max_slots)` per-arch
    /// path (the C2d §6.1.22 surface).
    #[test]
    fn h96_qwen35_and_qwen3vl_surfaces_unchanged() {
        let src = include_str!("engine.rs");
        // Find the Qwen35LoadedModel struct definition.
        // Qwen35LoadedModel is defined in engine_qwen35.rs (sibling
        // module); the engine.rs file only references it. Pin: no
        // Gemma 4 hybrid field name leaked into Qwen35 surface.
        let qwen_src = include_str!("engine_qwen35.rs");
        assert!(
            !qwen_src.contains("multi_seq_kv_hybrid"),
            "H96 FALSIFIED: `multi_seq_kv_hybrid` field name leaked \
             into engine_qwen35.rs. iter-C2c-cont is a Gemma 4-only \
             extension; Qwen35 has its own HybridKvCache multi-seq \
             surface (C2d §6.1.22)."
        );
        assert!(
            !qwen_src.contains("Gemma4HybridSlotAwareProvisionFailed"),
            "H96 FALSIFIED: `Gemma4HybridSlotAwareProvisionFailed` \
             variant referenced inside engine_qwen35.rs. Per-family \
             discriminants must stay per-family."
        );
        // Qwen35SlotAwareProvisionFailed Display contract preserved.
        assert!(
            src.contains("Qwen35SlotAwareProvisionFailed {"),
            "H96 FALSIFIED: `Qwen35SlotAwareProvisionFailed` variant \
             removed by iter-C2c-cont (which is supposed to be a \
             Gemma 4-only additive surface)."
        );
        // The hybrid allocator IS imported in this file's
        // `provision_multi_seq_kv_for_slot_aware` (Gemma 4) but is
        // NOT referenced anywhere ELSE in engine.rs (no orphan ref).
        // Acceptable references: the use-stmt inside the fn body + the
        // call site itself + this test module's docstring.
        // Pin: the allocator is NOT called from any other fn body.
        // Source-grep across the file's other fns.
        // Specifically: NOT in the spawn_with_mode Qwen35 arm.
        let qwen35_arm_marker = "LoadedModel::Qwen35(mut q) => {";
        if let Some(qwen_arm_idx) = src.find(qwen35_arm_marker) {
            let qwen_arm_window = &src[qwen_arm_idx..(qwen_arm_idx + 6000).min(src.len())];
            assert!(
                !qwen_arm_window.contains("alloc_multi_seq_hybrid_kv_for_layer"),
                "H96 FALSIFIED: Qwen35 spawn-arm body calls \
                 `alloc_multi_seq_hybrid_kv_for_layer` — that's a \
                 Gemma 4 module primitive. Cross-family leakage."
            );
            assert!(
                !qwen_arm_window.contains("multi_seq_kv_hybrid"),
                "H96 FALSIFIED: Qwen35 spawn-arm references \
                 `multi_seq_kv_hybrid` (the Gemma 4 field). Per-family \
                 surface segregation broken."
            );
        }
        // ADR-040 iter-C2e (2026-05-30 §6.1.52) — Qwen3VL SlotAware
        // spawn-arm FLIPPED to `Ok(Engine)` via the witness-only
        // provisioner (mirror of C2d for Qwen35). Sibling discipline
        // pin (REVISED post-C2e): iter-C2c-cont (Gemma 4 hybrid
        // scaffold) must not have FLIPPED the Qwen3VL arm itself —
        // C2c-cont ships strictly inside the Gemma 4 spawn-arm body.
        // The C2e flip is a SEPARATE iter that ships the Qwen3VL arm
        // body containing `Qwen3VLSlotAwareProvisionFailed` + the
        // `spawn_inner_with_slot_aware` delegate.
        let qwen3vl_c2e_marker = "Qwen3VLSlotAwareProvisionFailed";
        assert!(
            src.contains(qwen3vl_c2e_marker),
            "H96 FALSIFIED (post-C2e revision per §6.1.52): Qwen3VL \
             SlotAware spawn arm no longer references the C2e typed-error \
             variant `Qwen3VLSlotAwareProvisionFailed`. iter-C2c-cont must \
             NOT regress the C2e Qwen3VL spawn-arm flip."
        );
    }

    /// **H91-extension (skip-mode)** — Sub-deferral runbook pointer:
    /// the iter-C2c-cont closure ADR block (§6.1.33) NAMES the
    /// downstream iter-2B kernel-dispatch refactor that consumes the
    /// new field. Pin against an iter-C2c-cont commit that lands the
    /// scaffold without naming the next iter.
    #[test]
    fn h91_extension_iter_c2c_cont_names_downstream_iter2b() {
        let adr = crate::serve::api::engine::adr040_history_doc() /* iter-230 A1: §6.1.x moved to history (aeb6e87c) */;
        assert!(
            adr.contains("### 6.1.33"),
            "H91-ext FALSIFIED: ADR-040 §6.1.33 closure block not \
             found. iter-C2c-cont landing without a closure block \
             violates the §6.1.N-per-iter discipline."
        );
        let closure_marker = "### 6.1.33";
        let closure_start = adr
            .find(closure_marker)
            .expect("H91-ext: §6.1.33 marker missing (asserted above)");
        let closure_end_off = adr[closure_start..]
            .find("\n### ")
            .or_else(|| adr[closure_start..].find("\n---\n"))
            .or_else(|| adr[closure_start..].find("\n## "))
            .unwrap_or_else(|| adr[closure_start..].len().min(40_000));
        let closure_body = &adr[closure_start..closure_start + closure_end_off];
        for required in [
            "iter-C2c-cont",
            "MultiSeqHybridKvBuffers",
            "iter-B4c-kernel-iter-2B",
            "H10",
        ] {
            assert!(
                closure_body.contains(required),
                "H91-ext FALSIFIED: ADR-040 §6.1.33 closure body does \
                 NOT name `{required}`. Sub-deferral runbook incomplete."
            );
        }
    }
}

// ============================================================================
// ADR-040 iter-B4c-kernel iter-2B — H97-H103 hypothesis pins
// ============================================================================
//
// Scope (iter-2B wires the actual HybridKvBuffers slot routing through
// `forward_prefill_with_soft_tokens_slot_aware`'s `INVESTIGATION_ENV.hybrid_kv`
// dispatch-fork branch — the production-engagement sub-iter per H10 falsification
// at §6.1.11 (HF2Q_HYBRID_KV is default-true since ADR-029 iter-13, 2026-05-11).
//
//   * iter-2A (§6.1.32, commit `6a5b7ca4`) shipped:
//     - NEW `forward_prefill_with_soft_tokens_slot_aware` fn signature + bounds-
//       first preflight + 4-way dispatch fork; every branch surfaces typed
//       `MultiSeqError::CapabilityUnsupported`.
//   * iter-C2c-cont (§6.1.33, commit `ec7b7594`) shipped:
//     - NEW `GemmaLoadedModel.multi_seq_kv_hybrid: Option<Vec<MultiSeqHybridKvBuffers>>`
//       sibling field + spawn-time provisioning gated on `INVESTIGATION_ENV.hybrid_kv`.
//
//   * iter-B4c-kernel iter-2B (THIS commit, §6.1.34) ships:
//     - EXTENDED new fn signature with `multi_seq_kv_hybrid: Option<&mut
//       Vec<MultiSeqHybridKvBuffers>>` (additive parameter; HB scaffold param
//       preserved verbatim).
//     - REPLACED the iter-2A `INVESTIGATION_ENV.hybrid_kv` typed-error branch
//       body with real slot routing: per-layer slot-view construction via
//       `MlxBuffer::slice_view(byte_offset, n_elements) + .with_shape([nkv, cap,
//       hd])`, mount on `self.hybrid_kv`, delegate to
//       `forward_prefill_with_soft_tokens_resume`, restore prior value on exit.
//     - ALIGNED the sibling fn's lazy-alloc gate at line 842 with the decode-
//       path gate at `forward_gpu.rs:413` (add `&& self.hybrid_kv.is_none()`)
//       so the slot-view mount is not obliterated by the sibling fn's
//       unconditional rebuild. Decode-path precedent (forward_gpu.rs:413)
//       proves the gate is consistent with prior-art Gemma 4 behavior; SerialFifo
//       byte-equivalence preserved because SerialFifo enters with
//       `self.hybrid_kv == None` (gate fires identically).
//     - EXTENDED orchestrator `generate_gemma4_once_slot_aware` signature with
//       `multi_seq_kv_hybrid: Option<&mut Vec<MultiSeqHybridKvBuffers>>` +
//       per-layer entry+exit `reset_for_slot` on it parallel to HB scaffold.
//     - EXTENDED worker arm with parallel take/restore on both
//       `g.multi_seq_kv` AND `g.multi_seq_kv_hybrid` scaffolds.
//     - Sub-deferred xlen BF16 K/V slot routing as `iter-B4c-kernel-iter-2B-xlen`
//       (typed `CapabilityUnsupported` when any layer's `bf16_xlen_k.is_some()` —
//       gated on `HF2Q_DFLASH_XLEN_SDPA=1` opt-in surface).
//
// Tests (H97-H103):
//   H97 (skip-mode): The iter-2A hybrid-branch typed-error label
//                    `gemma4-forward-prefill-slot-N-hybrid (iter-B4c-kernel-iter-2B per`
//                    is REMOVED from the new fn body. Positive pin: the slot-view
//                    mount IS present (slice_view + with_shape pattern).
//   H98 (skip-mode): New fn signature gains the `multi_seq_kv_hybrid:
//                    Option<&mut Vec<MultiSeqHybridKvBuffers>>` parameter
//                    (additive — `multi_seq_kv_hb: &mut Vec<MultiSeqHbKvBuffers>`
//                    preserved verbatim per H84).
//   H99 (skip-mode): Orchestrator `generate_gemma4_once_slot_aware` signature
//                    gains the parallel `multi_seq_kv_hybrid` parameter + the
//                    body threads it via take+restore through the worker arm.
//   H100 (skip-mode): The slot-view mount uses the per-slot byte offset shape
//                     `slot_id.0 ... nkv * cap * hd * 2` (F16 K is 2 bytes/elem)
//                     mirroring Qwen35 B4a-cont's slice_view pattern per §6.1.5.
//   H101 (per-slot isolation): NOT runnable without model load. Replaced by a
//                     unit test on the slot-view mount primitive: building slot
//                     1's view from a multi-seq buffer produces an MlxBuffer at
//                     the correct byte_offset (slot 1's region byte-isolated
//                     from slot 0's region).
//   H102 (skip-mode): SerialFifo byte-equivalence preserved — the sibling fn
//                     `forward_prefill_with_soft_tokens_resume`'s SIGNATURE is
//                     unchanged (H86 PRESERVED); the lazy-alloc body change at
//                     line 842 is gated on `self.hybrid_kv.is_none()` matching
//                     decode-path discipline at `forward_gpu.rs:413` (gate
//                     fires identically when entering with `None`).
//   H103 (skip-mode): HbKvBuffers regime UNCHANGED — the
//                     `iter-B4c-kernel-iter-2A-cont` typed deferral on the HB-
//                     encoded branch is still surfaced verbatim (iter-2A-cont
//                     remains pending). Defends against an iter-2B commit that
//                     accidentally collapses the HB branch.
// ----------------------------------------------------------------------------

#[cfg(test)]
mod adr040_phase_b_iter_b4c_kernel_iter2b_gemma4_tests {
    // Skip-mode source-grep tests; intentionally NO `use super::*;`
    // for the source-grep helpers.

    /// **H97 (skip-mode)** — The iter-2A hybrid-branch typed-error label
    /// is REPLACED with real slot routing.  iter-2A surfaced
    /// `MultiSeqError::CapabilityUnsupported { capability: "gemma4-forward-
    /// prefill-slot-N-hybrid (iter-B4c-kernel-iter-2B per ADR-040 §6.1.32 ..." }`
    /// at every entry into the `INVESTIGATION_ENV.hybrid_kv` branch;
    /// iter-2B REMOVES that typed error from the branch body (the
    /// production-engagement code path now does real work).
    ///
    /// Positive pin: the slot-view mount via `slice_view` IS present —
    /// the load-bearing primitive for per-slot routing per §6.1.5.
    #[test]
    fn h97_iter2a_hybrid_branch_typed_error_replaced_with_slot_routing() {
        let src = include_str!("../forward_prefill.rs");
        let fn_marker = "pub fn forward_prefill_with_soft_tokens_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H97: new fn marker present (H84 asserts)");
        // ADR-040 iter-2C + iter-2D (§6.1.46) — window bumped from
        // 30_000 to 80_000 to cover both the hybrid + HB-encoded
        // branches now that the dense F32 + legacy 4-bit branches
        // sit before them.
        let fn_window = &src[fn_idx..(fn_idx + 80_000).min(src.len())];
        // The iter-2A hybrid-branch typed-error label is the literal
        // `"gemma4-forward-prefill-slot-N-hybrid (iter-B4c-kernel-iter-2B per"`
        // (the iter-2A pin). iter-2B REMOVES it (real routing).
        let iter2a_hybrid_label =
            "gemma4-forward-prefill-slot-N-hybrid (iter-B4c-kernel-iter-2B per";
        assert!(
            !fn_window.contains(iter2a_hybrid_label),
            "H97 FALSIFIED: new fn body still contains iter-2A hybrid \
             branch typed-error label `{iter2a_hybrid_label}`. iter-2B \
             slot routing NOT landed — production-default request still \
             surfaces CapabilityUnsupported at the hybrid_kv branch."
        );
        // Positive pin: the slot-view mount via slice_view IS present —
        // load-bearing per-slot routing primitive per §6.1.5.
        assert!(
            fn_window.contains(".slice_view("),
            "H97 FALSIFIED: new fn body does NOT contain `.slice_view(` \
             — the slot-view mount primitive is missing. Per-slot \
             routing through HybridKvBuffers' slot region cannot work."
        );
    }

    /// **H98 (skip-mode)** — New fn signature gains the
    /// `multi_seq_kv_hybrid: Option<&mut Vec<MultiSeqHybridKvBuffers>>`
    /// parameter (additive — iter-2A's `multi_seq_kv_hb: &mut Vec<
    /// MultiSeqHbKvBuffers>` parameter is preserved verbatim per H84).
    ///
    /// Pin defends a regression where iter-2B accidentally REPLACES
    /// the HB scaffold param (would break H84 + the iter-2A-cont
    /// follow-up's destination).
    #[test]
    fn h98_new_fn_signature_extended_with_multi_seq_kv_hybrid_param() {
        let src = include_str!("../forward_prefill.rs");
        let fn_marker = "pub fn forward_prefill_with_soft_tokens_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H98: new fn marker present (H84 asserts)");
        let sig_window = &src[fn_idx..(fn_idx + 2500).min(src.len())];
        // H84: HB scaffold param preserved verbatim.
        assert!(
            sig_window.contains("multi_seq_kv_hb: &mut Vec<MultiSeqHbKvBuffers>"),
            "H98 FALSIFIED: iter-2A HB scaffold parameter \
             `multi_seq_kv_hb: &mut Vec<MultiSeqHbKvBuffers>` is no \
             longer present — H84 + iter-2A-cont follow-up's \
             destination accidentally removed."
        );
        // iter-2B ADD: hybrid scaffold parameter.
        assert!(
            sig_window.contains("multi_seq_kv_hybrid:"),
            "H98 FALSIFIED: new fn signature missing \
             `multi_seq_kv_hybrid:` parameter. iter-2B production-default \
             slot routing has no scaffold to consume — HF2Q_HYBRID_KV \
             branch cannot land per-slot K/V writes."
        );
        // Specific type shape — Option wrapping per the iter-C2c-cont
        // field type so a SlotAware engine with HF2Q_HYBRID_KV=0 can
        // pass `None` without panic.
        assert!(
            sig_window.contains("Option<&mut Vec<MultiSeqHybridKvBuffers>>"),
            "H98 FALSIFIED: new fn signature's `multi_seq_kv_hybrid` \
             param is not `Option<&mut Vec<MultiSeqHybridKvBuffers>>`. \
             iter-C2c-cont's field is `Option<Vec<_>>` (None when \
             HF2Q_HYBRID_KV=0); the param type must match for clean \
             take-and-restore at the worker arm."
        );
    }

    /// **H99 (skip-mode)** — Orchestrator
    /// `generate_gemma4_once_slot_aware` signature gains the parallel
    /// `multi_seq_kv_hybrid: Option<&mut Vec<MultiSeqHybridKvBuffers>>`
    /// parameter + the worker arm threads it via take+restore on
    /// `g.multi_seq_kv_hybrid` parallel to `g.multi_seq_kv`.
    ///
    /// Mirrors iter-1's H79 pin (take-and-restore) for the new sibling
    /// field iter-C2c-cont provisioned.
    #[test]
    fn h99_orchestrator_threads_multi_seq_kv_hybrid_via_take_restore() {
        let src = include_str!("engine.rs");
        let fn_marker = "fn generate_gemma4_once_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H99: generate_gemma4_once_slot_aware not found");
        let fn_window = &src[fn_idx..(fn_idx + 10_000).min(src.len())];
        // Orchestrator signature gains the hybrid scaffold param.
        assert!(
            fn_window.contains("multi_seq_kv_hybrid:"),
            "H99 FALSIFIED: orchestrator signature missing \
             `multi_seq_kv_hybrid:` parameter. Hybrid scaffold cannot \
             be threaded from worker arm to the new fn — iter-2B \
             slot routing broken at the orchestrator boundary."
        );
        // Worker arm must take + restore g.multi_seq_kv_hybrid.
        let worker_marker = "g.multi_seq_kv_hybrid.take()";
        assert!(
            src.contains(worker_marker),
            "H99 FALSIFIED: worker arm does NOT call \
             `g.multi_seq_kv_hybrid.take()`. The persistent hybrid \
             scaffold iter-C2c-cont provisioned is not consumed — \
             every request at SlotId(N>0) would surface defense-in-depth \
             `None` instead of using the field."
        );
        let restore_marker = "g.multi_seq_kv_hybrid = Some(";
        assert!(
            src.contains(restore_marker),
            "H99 FALSIFIED: worker arm does NOT restore \
             `g.multi_seq_kv_hybrid` via `Some(_)` after the call. The \
             next request on the same slot would find `None` and \
             defense-in-depth-fail."
        );
    }

    /// **H100 (skip-mode)** — The slot-view mount uses the per-slot byte
    /// offset for the F16 K buffer following Qwen35 B4a-cont's
    /// slice_view pattern per §6.1.5.  F16 = 2 bytes/elem, so byte
    /// offset = `slot_id.0 * nkv * cap * hd * 2`.
    ///
    /// Pin defends a slot-routing regression where iter-2B accidentally
    /// uses a different byte-size multiplier (e.g., 4 for F32) — silently
    /// routes to the WRONG slot's region.
    #[test]
    fn h100_slot_view_byte_offset_uses_f16_2_byte_multiplier() {
        let src = include_str!("../forward_prefill.rs");
        let fn_marker = "pub fn forward_prefill_with_soft_tokens_slot_aware(";
        let fn_idx = src.find(fn_marker).expect("H100: new fn marker present");
        // ADR-040 iter-2C + iter-2D (§6.1.46) — window bumped from
        // 30_000 to 80_000 to cover both the hybrid + HB-encoded
        // branches now that the dense F32 + legacy 4-bit branches
        // sit before them.
        let fn_window = &src[fn_idx..(fn_idx + 80_000).min(src.len())];
        // The F16 K byte size discipline — 2 bytes/elem.  Honest match:
        // either `* 2` arithmetic on a `nkv * cap * hd` term OR the
        // dtype-aware `DType::F16.size_of()` lookup.  We accept either
        // form — the load-bearing invariant is the F16 byte-size
        // multiplier appears in the slot-view byte-offset arithmetic.
        let has_explicit_2 = fn_window.contains("* 2)")
            || fn_window.contains("* 2 ")
            || fn_window.contains("(2u64)")
            || fn_window.contains("size_of::<u16>()");
        let has_dtype_lookup = fn_window.contains("DType::F16.size_of()");
        assert!(
            has_explicit_2 || has_dtype_lookup,
            "H100 FALSIFIED: new fn body does NOT use a 2-byte multiplier \
             on the slot-view byte-offset arithmetic. F16 K's per-slot \
             byte offset should be `slot_id.0 * nkv * cap * hd * 2`; \
             slot routing would silently target the WRONG slot's region."
        );
        // Pin the slot_id.0 multiplier explicitly: byte offset MUST
        // include `slot_id.0` as a factor (else every slot routes to
        // slot 0's region — the iter-1 H77 failure mode).
        assert!(
            fn_window.contains("slot_id.0") || fn_window.contains("slot_id . 0"),
            "H100 FALSIFIED: new fn body does NOT reference `slot_id.0` \
             in the slot-view byte-offset arithmetic. Per-slot routing \
             is broken — every slot would target slot 0's region."
        );
    }

    /// **H101 (skip-mode + structural)** — Per-slot isolation: the
    /// slot-view mount applies `slice_view(byte_offset, n_elements)
    /// + .with_shape([nkv, cap, hd])` so the per-slot region is a
    /// 3-D view at the per-slot byte offset (legacy `HybridKvBuffers`
    /// shape preserved).
    ///
    /// Mirrors A3b iter-1.5 H12's "write to slot 0 leaves slot 1
    /// byte-zero" discipline at the byte-layout level — the slot-view's
    /// underlying ARC handle is the same, the byte offset distinguishes
    /// the per-slot region.
    #[test]
    fn h101_slot_view_preserves_legacy_hybrid_kv_buffers_3d_shape() {
        let src = include_str!("../forward_prefill.rs");
        let fn_marker = "pub fn forward_prefill_with_soft_tokens_slot_aware(";
        let fn_idx = src.find(fn_marker).expect("H101: new fn marker present");
        // ADR-040 iter-2C + iter-2D (§6.1.46) — window bumped from
        // 30_000 to 80_000 to cover both the hybrid + HB-encoded
        // branches now that the dense F32 + legacy 4-bit branches
        // sit before them.
        let fn_window = &src[fn_idx..(fn_idx + 80_000).min(src.len())];
        // The `with_shape` call IS present (shape-preserving view).
        assert!(
            fn_window.contains(".with_shape("),
            "H101 FALSIFIED: new fn body does NOT contain \
             `.with_shape(` — the legacy `HybridKvBuffers` 3-D shape \
             `[nkv, capacity, head_dim]` is not preserved. \
             Downstream kernels reading `.shape()` for stride math \
             would see the slice_view's flat 1-D shape and miscompute."
        );
        // The legacy HybridKvBuffers struct construction IS present
        // (the mount path constructs `HybridKvBuffers { k, v_packed,
        // v_norms, ... }` from the slot-views).  Mirrors `alloc_hybrid_kv_for_layer`
        // at gemma4/kv_cache.rs:740.
        assert!(
            fn_window.contains("HybridKvBuffers {"),
            "H101 FALSIFIED: new fn body does NOT construct \
             `HybridKvBuffers {{ ... }}` from the slot-views. The mount \
             path must produce the legacy struct so the sibling fn's \
             `if let Some(ref hybrid_kv) = self.hybrid_kv` consumer at \
             line 1293 can read it bit-identically."
        );
    }

    /// **H102 (skip-mode)** — SerialFifo byte-equivalence preserved.
    ///
    /// The sibling fn `forward_prefill_with_soft_tokens_resume`'s
    /// SIGNATURE is unchanged (H86 PRESERVED).  The body change at
    /// line ~842 adds `&& self.hybrid_kv.is_none()` matching the
    /// decode-path gate at `forward_gpu.rs:413`.
    ///
    /// SerialFifo enters the sibling fn with `self.hybrid_kv == None`
    /// (no prior call mounted a slot-view), so the gate fires
    /// identically → byte-equivalent allocation behavior.
    #[test]
    fn h102_serial_fifo_byte_equivalence_preserved_via_decode_aligned_gate() {
        let pf_src = include_str!("../forward_prefill.rs");
        // (a) H86 PRESERVED: sibling fn signature unchanged.
        let sibling_marker = "pub fn forward_prefill_with_soft_tokens_resume(";
        let sibling_idx = pf_src
            .find(sibling_marker)
            .expect("H102: sibling fn marker present (H86 asserts)");
        let sibling_sig = &pf_src[sibling_idx..(sibling_idx + 600).min(pf_src.len())];
        // Sibling signature MUST NOT contain `slot_id` or `multi_seq_kv*`.
        assert!(
            !sibling_sig.contains("slot_id"),
            "H102 FALSIFIED: sibling fn signature now contains `slot_id` \
             — iter-2B accidentally modified the sibling fn signature. \
             H86 / H1 / H2 / H23 / H41 / H44 byte-equivalence chain \
             broken."
        );
        assert!(
            !sibling_sig.contains("multi_seq_kv"),
            "H102 FALSIFIED: sibling fn signature now contains \
             `multi_seq_kv*` — iter-2B accidentally modified the \
             sibling fn signature. H86 / H1 / H2 / H23 / H41 / H44 \
             byte-equivalence chain broken."
        );
        // (b) Lazy-alloc gate aligns with decode-path discipline.
        // Find the line that contains `INVESTIGATION_ENV.hybrid_kv` in
        // the alloc block (line ~842).  iter-2B adds the
        // `&& self.hybrid_kv.is_none()` predicate matching
        // `forward_gpu.rs:413`.
        let alloc_marker = "[ADR-028 Phase 10c] Allocating hybrid_kv";
        let alloc_idx = pf_src
            .find(alloc_marker)
            .expect("H102: alloc-site eprintln marker not found");
        // Pull a 200-char window BEFORE the eprintln (covers the
        // `if INVESTIGATION_ENV.hybrid_kv ...` predicate line).
        let pre_alloc_window = &pf_src[alloc_idx.saturating_sub(400)..alloc_idx];
        assert!(
            pre_alloc_window.contains("self.hybrid_kv.is_none()"),
            "H102 FALSIFIED: prefill alloc gate at line ~842 does NOT \
             check `self.hybrid_kv.is_none()`. The slot-view mount \
             from iter-2B would be obliterated by the sibling fn's \
             unconditional rebuild. Decode-path precedent at \
             forward_gpu.rs:413 already uses this gate; iter-2B aligns \
             prefill with decode."
        );
    }

    /// **H103 (skip-mode)** — HbKvBuffers regime UNCHANGED.  The
    /// iter-2A-cont sub-deferral on the HB-encoded branch (HF2Q_HYBRID_KV=0
    /// opt-out surface) is still surfaced verbatim.  Defends against an
    /// iter-2B commit that accidentally collapses BOTH HB and Hybrid
    /// branches into one.
    #[test]
    fn h103_hb_encoded_branch_iter2a_cont_typed_deferral_preserved() {
        let src = include_str!("../forward_prefill.rs");
        let fn_marker = "pub fn forward_prefill_with_soft_tokens_slot_aware(";
        let fn_idx = src.find(fn_marker).expect("H103: new fn marker present");
        // ADR-040 iter-2C + iter-2D (§6.1.46) — window bumped from
        // 30_000 to 80_000 to cover both the hybrid + HB-encoded
        // branches now that the dense F32 + legacy 4-bit branches
        // sit before them.
        let fn_window = &src[fn_idx..(fn_idx + 80_000).min(src.len())];
        // The iter-2A-cont typed-deferral label MUST still be present
        // verbatim (HB-encoded branch remains pending).
        let iter2a_cont_label = "iter-B4c-kernel-iter-2A-cont per ADR-040 §6.1.32";
        assert!(
            fn_window.contains(iter2a_cont_label),
            "H103 FALSIFIED: HB-encoded branch typed-deferral label \
             `{iter2a_cont_label}` REMOVED. iter-2A-cont is NOT in \
             scope of iter-2B; the HF2Q_HYBRID_KV=0 opt-out path \
             would now silently surface no error or wrong routing."
        );
        // Also pins iter-2C + iter-2D sub-deferrals preserved (the 4-way
        // dispatch fork shape is intact).
        for label in [
            "iter-B4c-kernel-iter-2C per ADR-040 §6.1.32",
            "iter-B4c-kernel-iter-2D per ADR-040 §6.1.32",
        ] {
            assert!(
                fn_window.contains(label),
                "H103 FALSIFIED: dispatch-fork sub-deferral `{label}` \
                 REMOVED. iter-2B accidentally collapsed the 4-way \
                 dispatch fork — non-default KV regimes lose their \
                 typed-error labels."
            );
        }
        // NEW iter-2B-xlen sub-deferral IS named (xlen BF16 K/V slot
        // routing carved out for a future iter).
        assert!(
            fn_window.contains("iter-B4c-kernel-iter-2B-xlen"),
            "H103 FALSIFIED: xlen BF16 K/V sub-deferral \
             `iter-B4c-kernel-iter-2B-xlen` is NOT named in the new fn \
             body. The HF2Q_DFLASH_XLEN_SDPA=1 opt-in surface has no \
             operator-grep'able pin pointing at the next iter."
        );
    }
}

// ============================================================================
// ADR-040 iter-B4c-kernel iter-3 — H104-H109 hypothesis pins
// ============================================================================
//
// Scope (iter-3 advances the Gemma 4 worker-arm lift arc by one arm —
// GenerateStream — direct mirror of Qwen35 iter-C2d-cont-kernel iter-2
// §6.1.28 for the streaming surface):
//
//   * iter-1 (§6.1.31, commit `bac4c385`) shipped the Generate-arm
//     scaffold lift onto the persistent multi-seq `MultiSeqHbKvBuffers`
//     + sibling `MultiSeqHybridKvBuffers` (`reset_for_slot` primitive +
//     `generate_gemma4_once_slot_aware` orchestrator + worker-arm
//     dispatch fork).
//   * iter-2A (§6.1.32) landed the model-level slot-aware fn
//     `forward_prefill_with_soft_tokens_slot_aware` with bounds-first
//     pre-flight + 4-way KV-regime dispatch fork.
//   * iter-2B (§6.1.34, commit `1676fcd1`) landed the production-default
//     hybrid F16-K + TQ-HB-V slot routing via `MultiSeqHybridKvBuffers`
//     slice_view mount + delegate-to-sibling pattern.
//
//   * iter-3 (THIS commit, ADR-040 §6.1.35) ships:
//     - NEW `generate_stream_gemma4_once_slot_aware` orchestrator at
//       engine.rs (mirror of iter-1's `generate_gemma4_once_slot_aware`
//       shape for the streaming-event-channel result surface).
//       Reuses iter-1's `MultiSeqHbKvBuffers::reset_for_slot` + sibling
//       `MultiSeqHybridKvBuffers::reset_for_slot` primitives + iter-2B's
//       `forward_prefill_with_soft_tokens_slot_aware` kernel call.
//     - `worker_run` Gemma 4 GenerateStream-arm lift fork (take +
//       restore on `g.multi_seq_kv` + `g.multi_seq_kv_hybrid`).
//     - Vision-augmented streaming deferral (soft_tokens.is_empty() ==
//       false surfaces typed error event citing iter-B4c-kernel-iter-5).
//     - Multi-token decode-loop body wrapping deferral (post-prefill
//       Ok branch surfaces typed error event citing
//       iter-B4c-kernel-iter-2-decode — same sub-deferral the iter-2B
//       Generate-arm IIFE surfaces).
//     - Tests revised for the post-iter-3 lifted state: H40, H56, H62,
//       H68, H75, H82 swap their `gemma4-forward-prefill-slot-N
//       (iter-C2c-cont` literal for the surviving Embed-arm label
//       `gemma4-forward-embed-last-slot-N (iter-C2c-cont` (sibling-
//       discipline intent preserved; the C2c clamp survives on Embed +
//       SoftTokens arms).
//
// Tests (H104-H109 mirror Qwen35 iter-2 H58-H63 1:1):
//   H104 (skip-mode): SerialFifo + SlotId(0) Gemma 4 GenerateStream
//                     dispatch byte-equivalent post-iter-3.  The
//                     `handle.slot_id != SlotId(0)` predicate short-
//                     circuits below the lift fork; the existing
//                     `generate_stream_once` dispatch at the `match
//                     &mut loaded` block fires verbatim.
//   H105 (skip-mode): iter-3 lift landed at GenerateStream arm: the
//                     worker_run body contains a real call to
//                     `generate_stream_gemma4_once_slot_aware(` under
//                     the Gemma 4 GenerateStream arm.
//   H106 (skip-mode): persistent multi_seq_kv + multi_seq_kv_hybrid
//                     take/restore at the iter-3 lift call site
//                     (both scaffolds; mirror of iter-2B Generate-
//                     arm take/restore pattern).
//   H107 (skip-mode): `reset_for_slot(slot_id)` at entry + exit on
//                     BOTH scaffolds (the slot-aware streaming fn body
//                     has ≥ 2 occurrences for the HB scaffold AND ≥ 2
//                     for the hybrid scaffold).
//   H108 (skip-mode): Qwen35 + Qwen3VL + Gemma 4 Generate (iter-1+
//                     2A+2B) preserved; Gemma 4 Embed (iter-4) +
//                     SoftTokens (iter-5) clamps still present.  The
//                     iter-3 narrowing reduces the surviving Gemma 4
//                     clamp surface from 3 arms (post-iter-1) to 2
//                     arms (Embed + SoftTokens).
//   H109 (skip-mode): SSE event ordering preserved.  iter-3 today
//                     emits typed `Error` events only (no Delta events
//                     because the decode loop is iter-2-decode scope);
//                     the `send!` macro is defined; the
//                     iter-2-decode typed-deferral substring is
//                     present in the slot-aware streaming fn body.
// ----------------------------------------------------------------------------

#[cfg(test)]
mod adr040_phase_b_iter_b4c_kernel_iter3_gemma4_tests {
    // Skip-mode source-grep tests; intentionally NO `use super::*;`
    // (tests rely on `include_str!` against engine.rs + the ADR doc).

    // ── Helper: snip worker_run body the same way iter-1/2 tests do ──
    fn worker_run_body(src: &str) -> &str {
        let body_start = src
            .find("fn worker_run(")
            .expect("iter-3: worker_run entry not found");
        let body_after = &src[body_start..];
        let body_end_off = body_after
            .find("\n// The worker thread for `LoadedModel::Qwen35` returns a sentinel error")
            .or_else(|| body_after.find("\n/// Worker-thread entry point"))
            .unwrap_or(body_after.len().min(200_000));
        &body_after[..body_end_off]
    }

    /// **H104 (skip-mode)** — SerialFifo + SlotId(0) AND SlotAware +
    /// SlotId(0) Gemma 4 GenerateStream dispatch is byte-equivalent
    /// post-iter-3.
    ///
    /// Source-grep pin: the iter-3 lift fork at the GenerateStream
    /// arm uses the predicate `handle.slot_id != SlotId(0)`. SerialFifo
    /// always hands out SlotId(0) (FifoSchedulerAdapter invariant);
    /// SlotAware's first request also gets SlotId(0). In both cases
    /// the predicate is FALSE → the lift block falls through to the
    /// existing `match &mut loaded { LoadedModel::Gemma(g) =>
    ///     generate_stream_once(g, ..) }` dispatch, byte-equivalent
    /// to pre-iter-3 + pre-C2c.
    ///
    /// Defends the H1 / H2 / H23 / H41 / H44 / H77 byte-equivalence
    /// chain extended to the Gemma 4 streaming surface.  Direct
    /// mirror of Qwen35 iter-C2d-cont-kernel iter-2 H58.
    #[test]
    fn h104_slot_id_0_gemma4_stream_routes_through_generate_stream_once_byte_equivalent() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        // The pre-iter-3 `generate_stream_once` dispatch must still be
        // reachable from the worker arm for the SlotId(0) fallback.
        assert!(
            body.contains("generate_stream_once(\n                            g,\n                            &prompt_tokens,")
                || body.contains("generate_stream_once(") ,
            "H104 FALSIFIED: post-iter-3 worker_run Gemma 4 \
             GenerateStream dispatch no longer routes through \
             `generate_stream_once` for SlotId(0). The iter-3 lift \
             fork must be ADDITIVE (sibling above the `match &mut \
             loaded` dispatch), NOT REPLACE the SerialFifo / SlotId(0) \
             path. SerialFifo + SlotId(0) byte-equivalence \
             (H1 / H2 / H77 chain) is BROKEN for the Gemma 4 streaming \
             arm."
        );

        // The lift fork predicate at the Gemma 4 worker arms must be
        // `matches!(loaded, LoadedModel::Gemma(_)) && handle.slot_id != SlotId(0)`
        // — the same shape iter-1 used for the Generate arm.  Pin: at
        // least TWO occurrences of the literal predicate in the
        // worker_run body (iter-1 Generate arm + iter-3 GenerateStream
        // arm; Embed + SoftTokens still use the predicate too for
        // their clamps).
        let predicate = "matches!(loaded, LoadedModel::Gemma(_)) && handle.slot_id != SlotId(0)";
        let n = body.matches(predicate).count();
        assert!(
            n >= 4,
            "H104 FALSIFIED: the iter-3 lift fork predicate \
             `{predicate}` must appear at least 4 times in worker_run \
             body (one for iter-1 Generate lift, one for iter-3 \
             GenerateStream lift, one each for iter-4 Embed + iter-5 \
             SoftTokens clamps). Got {n}. Drift here means the lift \
             may fire at SlotId(0) too, breaking byte-equivalence."
        );
    }

    /// **H105 (skip-mode)** — iter-3 GenerateStream-arm lift landed
    /// at `worker_run`: the slot-aware fn
    /// `generate_stream_gemma4_once_slot_aware` is called from the
    /// worker_run body at the Gemma 4 GenerateStream arm.  Source-grep
    /// pin.  Mirror of Qwen35 iter-2 H59.
    #[test]
    fn h105_iter3_lift_landed_for_gemma4_generate_stream_arm() {
        let src = include_str!("engine.rs");
        // The slot-aware fn is defined in this file.
        assert!(
            src.contains("fn generate_stream_gemma4_once_slot_aware("),
            "H105 FALSIFIED: `generate_stream_gemma4_once_slot_aware` \
             is NOT defined in engine.rs. iter-B4c-kernel iter-3 \
             production surface MISSING — orchestrator scaffold not \
             landed."
        );
        let body = worker_run_body(src);
        assert!(
            body.contains("generate_stream_gemma4_once_slot_aware("),
            "H105 FALSIFIED: worker_run does NOT call \
             `generate_stream_gemma4_once_slot_aware`. iter-3 lift is \
             not wired into the dispatch fork — the GenerateStream-arm \
             SlotId(N>0) routing is missing."
        );

        // The slot-aware fn passes `slot_id` (the SlotId from the
        // admit'd handle), NOT a hard-coded SlotId(0). Pin via
        // substring search inside the lift call block.
        let lift_block_start = body
            .find("generate_stream_gemma4_once_slot_aware(")
            .expect("H105: lift call site not found");
        let lift_block_end = body[lift_block_start..]
            .find(");")
            .map(|off| lift_block_start + off + 2)
            .unwrap_or(body.len().min(lift_block_start + 2000));
        let lift_block = &body[lift_block_start..lift_block_end];
        assert!(
            lift_block.contains("slot_id"),
            "H105 FALSIFIED: the lift call site does not pass `slot_id` \
             into `generate_stream_gemma4_once_slot_aware`. The iter-3 \
             lift must thread the admit'd SlotHandle's slot_id into \
             the slot-aware fn. Got block: {lift_block}"
        );
        assert!(
            !lift_block.contains(", SlotId(0),"),
            "H105 FALSIFIED: lift call site contains a hard-coded \
             `SlotId(0)` literal argument. Per-slot routing is broken \
             — the orchestrator must receive the admit'd handle's \
             SlotId verbatim."
        );
    }

    /// **H106 (skip-mode)** — persistent multi_seq_kv +
    /// multi_seq_kv_hybrid take+restore pattern at the iter-3 lift
    /// call site.  Mirror of Qwen35 iter-2 H60 with both scaffolds.
    ///
    /// Pin both `g.multi_seq_kv.take()` + `g.multi_seq_kv_hybrid.take()`
    /// AND the corresponding restores after the call.  The take+restore
    /// pattern is required for:
    /// (a) two-iter symmetry — iter-1 already established this pattern
    ///     for the Generate arm with BOTH scaffolds (iter-2B); iter-3
    ///     must use the same shape so the persistent-scaffold
    ///     invariants hold across BOTH Generate + GenerateStream
    ///     requests at any slot.
    /// (b) defense against the same regressions H79 + H99 catch —
    ///     forgotten put-back → next request finds `is_none()`
    ///     defense-in-depth typed error; clone-instead-of-take →
    ///     persistent scaffold's per-slot state is not actually
    ///     mutated, defeating cross-request isolation.
    ///
    /// iter-3's take+restore is ADDITIVE — the worker_run body has
    /// BOTH the iter-1+2B Generate arm AND the iter-3 GenerateStream
    /// arm take+restores.  Pin via count ≥ 2 for both take and restore
    /// on the HB scaffold, and ≥ 2 on the hybrid scaffold.
    #[test]
    fn h106_lift_call_site_takes_and_restores_both_scaffolds() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        // HB scaffold take pattern (mirror of iter-1 H79).
        let hb_take_count = body.matches("g.multi_seq_kv.take()").count();
        assert!(
            hb_take_count >= 2,
            "H106 FALSIFIED: the iter-3 lift call site does not \
             `take()` the persistent HB scaffold out of \
             `GemmaLoadedModel.multi_seq_kv`. Expected at least 2 \
             occurrences of `g.multi_seq_kv.take()` in worker_run body \
             (one each for iter-1 Generate + iter-3 GenerateStream lift \
             forks); got {hb_take_count}."
        );
        // HB scaffold restore pattern.
        let hb_restore_count = body.matches("g.multi_seq_kv = Some(multi_seq)").count();
        assert!(
            hb_restore_count >= 2,
            "H106 FALSIFIED: the iter-3 lift call site does not put \
             the persistent HB scaffold back into `g.multi_seq_kv` \
             after the slot-aware streaming fn returns. Expected at \
             least 2 occurrences of `g.multi_seq_kv = Some(multi_seq)` \
             in worker_run body (one each for iter-1 + iter-3 lift \
             forks); got {hb_restore_count}. The next request to land \
             at SlotId(N>0) would find `multi_seq_kv.is_none()` and \
             hit the defense-in-depth typed error."
        );
        // Hybrid scaffold take pattern (mirror of iter-2B H99).
        let hyb_take_count = body.matches("g.multi_seq_kv_hybrid.take()").count();
        assert!(
            hyb_take_count >= 2,
            "H106 FALSIFIED: the iter-3 lift call site does not \
             `take()` the persistent hybrid scaffold out of \
             `GemmaLoadedModel.multi_seq_kv_hybrid`. Expected at least \
             2 occurrences of `g.multi_seq_kv_hybrid.take()` in \
             worker_run body (one each for iter-2B Generate + iter-3 \
             GenerateStream lift forks); got {hyb_take_count}. \
             Production-default hybrid path (HF2Q_HYBRID_KV=1 per H10) \
             would silently surface the iter-2A hybrid typed error or \
             route to slot 0's region."
        );
        // Hybrid scaffold restore pattern.
        let hyb_restore_count = body
            .matches("g.multi_seq_kv_hybrid = multi_seq_hybrid")
            .count();
        assert!(
            hyb_restore_count >= 2,
            "H106 FALSIFIED: the iter-3 lift call site does not put \
             the persistent hybrid scaffold back into \
             `g.multi_seq_kv_hybrid` after the slot-aware streaming \
             fn returns. Expected at least 2 occurrences of \
             `g.multi_seq_kv_hybrid = multi_seq_hybrid` in worker_run \
             body (one each for iter-2B + iter-3 lift forks); got \
             {hyb_restore_count}."
        );
    }

    /// **H107 (skip-mode)** — per-slot reset at entry + exit of the
    /// slot-aware streaming fn on BOTH scaffolds (HB + hybrid).
    /// Mirror of Qwen35 iter-2 H61 with the dual-scaffold discipline
    /// iter-1 established for Gemma 4.
    ///
    /// Pin: ≥ 2 occurrences of `reset_for_slot(slot_id)` inside the
    /// slot-aware streaming fn body for the HB scaffold (entry + exit
    /// via `multi_seq_kv.iter_mut()`), AND ≥ 2 occurrences inside the
    /// hybrid `if let Some(ref mut hybrid_scaffold) = ...` blocks
    /// (entry + exit when the hybrid Option is Some).
    #[test]
    fn h107_slot_aware_stream_fn_calls_reset_for_slot_at_entry_and_exit() {
        // reset_for_slot primitives (iter-1's load-bearing add).
        let kv_src = include_str!("../../../src/inference/models/gemma4/kv_cache.rs");
        assert!(
            kv_src.contains("pub fn reset_for_slot("),
            "H107 FALSIFIED: `reset_for_slot` is not defined in \
             gemma4/kv_cache.rs. iter-3 inherits this primitive from \
             iter-1; if it's gone, iter-1 was reverted."
        );

        let src = include_str!("engine.rs");
        let fn_marker = "fn generate_stream_gemma4_once_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H107: generate_stream_gemma4_once_slot_aware not defined");
        // Window covering the fn body — bound by next top-level fn
        // marker or end-of-file.
        let body_after = &src[fn_idx..];
        let body_end_off = body_after[fn_marker.len()..]
            .find("\nfn ")
            .map(|off| off + fn_marker.len())
            .unwrap_or(body_after.len().min(60_000));
        let fn_body = &body_after[..body_end_off];

        // Total reset_for_slot count ≥ 4: entry + exit on HB scaffold
        // (2) + entry + exit on hybrid scaffold (2 — wrapped in
        // Option Some-guards).
        let reset_calls = fn_body.matches("reset_for_slot(slot_id)").count();
        assert!(
            reset_calls >= 4,
            "H107 FALSIFIED: \
             `generate_stream_gemma4_once_slot_aware` must call \
             `reset_for_slot(slot_id)` at LEAST 4 times (entry + exit \
             on the HB scaffold via `multi_seq_kv.iter_mut()` + entry \
             + exit on the hybrid scaffold via `if let Some(ref mut \
             hybrid_scaffold) = multi_seq_kv_hybrid` Option guards). \
             Got {reset_calls} call(s).  Drift here means cross-request \
             isolation is BROKEN on at least one scaffold."
        );

        // Per-layer iteration on HB scaffold.
        assert!(
            fn_body.contains("multi_seq_kv.iter_mut()"),
            "H107 FALSIFIED: slot-aware stream fn does NOT iterate \
             per-layer via `multi_seq_kv.iter_mut()`. Per-layer reset \
             coverage on HB scaffold is incomplete."
        );
        // Per-layer iteration on hybrid scaffold (inside Some-guard).
        assert!(
            fn_body.contains("hybrid_scaffold.iter_mut()"),
            "H107 FALSIFIED: slot-aware stream fn does NOT iterate \
             per-layer via `hybrid_scaffold.iter_mut()` inside the \
             hybrid Option Some-guard. Per-layer reset coverage on \
             the production-default hybrid scaffold is incomplete."
        );
    }

    /// **H108 (skip-mode)** — Qwen35 + Qwen3VL + Gemma 4 Generate /
    /// Embed / SoftTokens worker arms unchanged by iter-3.  Mirror of
    /// H82 extended for the iter-3 narrowing: iter-3 replaces the
    /// iter-3 Gemma 4 GenerateStream clamp, so only the iter-4 (Embed)
    /// + iter-5 (GenerateWithSoftTokens) clamps remain in the Gemma 4
    /// worker_run surface (the Generate arm is also lifted post-iter-1+
    /// 2A+2B).
    #[test]
    fn h108_other_worker_arms_unchanged_by_iter3() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        // Qwen35 lift fns from iter-C2d-cont-kernel iter-1/2/3/4 all
        // STILL called.
        for lift_fn in [
            "super::engine_qwen35::generate_qwen35_once_slot_aware(",
            "super::engine_qwen35::generate_stream_qwen35_once_extended_slot_aware(",
            "super::engine_qwen35::embed_qwen35_slot_aware(",
            "super::engine_qwen35::generate_qwen35_once_with_soft_tokens_slot_aware(",
        ] {
            assert!(
                body.contains(lift_fn),
                "H108 FALSIFIED: Qwen35 lift fn `{lift_fn}` is NOT \
                 called from worker_run. iter-3 must NOT regress any \
                 Qwen35 worker-arm lift (§6.1.27/28/29/30)."
            );
        }

        // iter-1 Gemma 4 Generate-arm lift fn STILL called (iter-3
        // must not regress iter-1).
        assert!(
            body.contains("generate_gemma4_once_slot_aware("),
            "H108 FALSIFIED: iter-1 Gemma 4 Generate lift fn \
             `generate_gemma4_once_slot_aware` is NOT called from \
             worker_run. iter-3 must NOT regress iter-1's Generate-arm \
             lift (§6.1.31)."
        );

        // ADR-040 iter-C2e (2026-05-30 §6.1.52) — Qwen3VL clamp
        // SHIPPED post-Gemma 4 iter-3. Sibling discipline pin:
        // Gemma 4 iter-3 must not REMOVE the C2e Qwen3VL clamp.
        assert!(
            body.contains(
                "matches!(loaded, LoadedModel::Qwen3VlText(_)) && handle.slot_id != SlotId(0)"
            ),
            "H108 FALSIFIED (post-C2e revision per §6.1.52): Qwen3VL \
             clamp missing from worker_run. iter-C2e SHIPPED 2026-05-30 \
             adds the Qwen3VL clamp at the four worker arms; Gemma 4 \
             iter-3 must NOT regress the C2e clamp."
        );

        // Post-iter-4 (§6.1.36, 2026-05-30): the Embed-arm
        // `gemma4-forward-embed-last-slot-N (iter-C2c-cont` clamp label
        // is REMOVED from worker_run (iter-4 legitimately lifted the
        // Embed arm via `embed_gemma4_slot_aware`).  H108's prior
        // assertion that the Embed clamp persisted reflected iter-3's
        // state; iter-4 legitimately lifts that arm and removes the
        // label.  The sibling-discipline intent ("iter-N did not
        // regress prior iters' lifts") is preserved by pinning the
        // iter-4 lift fn is called (below) + the surviving iter-5
        // SoftTokens clamp.
        assert!(
            body.contains("embed_gemma4_slot_aware("),
            "H108 FALSIFIED: iter-4 Gemma 4 Embed lift fn \
             `embed_gemma4_slot_aware` is NOT called from worker_run. \
             Post-iter-4 §6.1.36 the Embed arm must route through the \
             slot-aware orchestrator; lift fn must be wired."
        );

        // ADR-040 iter-B4c-kernel iter-5 (§6.1.37 — TERMINAL Gemma 4
        // worker-arm lift) REVISES H108: the SoftTokens clamp label
        // `gemma4-forward-prefill-with-soft-tokens-slot-N (iter-C2c-cont`
        // is LEGITIMATELY REMOVED by iter-5.  Sibling-discipline intent
        // preserved via the positive assertion that the iter-5 lift fn
        // is called from worker_run (mirror of iter-4's own H108
        // revision pattern that swapped the Embed clamp persisted
        // assertion for the lift-fn-present assertion).
        assert!(
            body.contains("generate_gemma4_once_with_soft_tokens_slot_aware("),
            "H108 FALSIFIED (post-iter-5 revision per §6.1.37): \
             Gemma 4 iter-5 TERMINAL SoftTokens lift fn \
             `generate_gemma4_once_with_soft_tokens_slot_aware` is \
             NOT called from worker_run. iter-3 must NOT regress \
             iter-5's §6.1.37 lift."
        );
        assert!(
            body.contains("iter-B4c-kernel-iter-5"),
            "H108 FALSIFIED: iter-5 sub-deferral cite \
             `iter-B4c-kernel-iter-5` missing from worker_run. \
             SoftTokens-arm clamp's typed-deferral discipline broken."
        );
    }

    /// **H109 (skip-mode)** — SSE event ordering preserved in the
    /// slot-aware streaming fn.  iter-3 today emits typed
    /// `GenerationEvent::Error` events only (NO `Delta` events because
    /// the multi-token decode-loop body wrapping is iter-2-decode
    /// scope); when iter-B4c-kernel-iter-2-decode lands, the per-token
    /// Delta emission loop + terminal Done event will be added.
    ///
    /// Source-grep pin on the slot-aware streaming fn's body:
    /// (a) the `send!` macro is defined (the SSE helper that calls
    ///     `events.blocking_send` + bumps cancellation_counter + early-
    ///     returns on client disconnect);
    /// (b) the iter-B4c-kernel-iter-2-decode typed-deferral substring
    ///     is present (operator-grep'able pin for the next sub-iter);
    /// (c) the slot-aware fn emits `GenerationEvent::Error` events
    ///     (defense against a refactor that switched to
    ///     `GenerationEvent::Done` or another variant).
    ///
    /// Mirror of Qwen35 iter-2 H63 for the Gemma 4 surface.
    #[test]
    fn h109_slot_aware_stream_fn_preserves_sse_event_ordering() {
        let src = include_str!("engine.rs");
        let fn_marker = "fn generate_stream_gemma4_once_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H109: generate_stream_gemma4_once_slot_aware not defined");
        let body_after = &src[fn_idx..];
        let body_end_off = body_after[fn_marker.len()..]
            .find("\nfn ")
            .map(|off| off + fn_marker.len())
            .unwrap_or(body_after.len().min(60_000));
        let fn_body = &body_after[..body_end_off];

        // (a) `send!` macro defined inside the fn (the SSE emit
        //     helper) — defense against iter-3 calling
        //     events.blocking_send without the cancellation-counter
        //     early-return wiring.
        assert!(
            fn_body.contains("macro_rules! send {"),
            "H109 FALSIFIED: \
             `generate_stream_gemma4_once_slot_aware` body does not \
             define the `send!` macro for SSE emission. The macro \
             must wrap every `events.blocking_send(...)` call to \
             early-return on client-disconnect — mirror of \
             `generate_stream_once` + Qwen35 \
             `generate_stream_qwen35_once_extended_slot_aware` shape."
        );

        // (b) iter-2-decode typed-deferral substring present (the
        //     iter-3 lift defers the multi-token decode loop to
        //     iter-2-decode per ADR-040 §6.1.35).
        assert!(
            fn_body.contains("iter-B4c-kernel-iter-2-decode"),
            "H109 FALSIFIED: \
             `generate_stream_gemma4_once_slot_aware` body does not \
             cite the `iter-B4c-kernel-iter-2-decode` sub-deferral. \
             The streaming multi-token decode loop body wrapping is \
             not pinned to its next-iter destination — operator log \
             greps cannot land on the right pin pointer."
        );

        // (c) Error events emitted via the typed `GenerationEvent::Error`
        //     variant (defense against a refactor that switched to a
        //     different event variant).
        assert!(
            fn_body.contains("GenerationEvent::Error("),
            "H109 FALSIFIED: \
             `generate_stream_gemma4_once_slot_aware` body does not \
             emit `GenerationEvent::Error(...)` events. Drift here \
             means the slot-aware streaming fn emits errors through a \
             different event variant — breaks SSE consumer parity \
             with the pre-iter-3 stream shape (the iter-3 typed \
             sub-deferrals MUST surface via this variant so the SSE \
             handler maps them to clean stream termination)."
        );

        // Exit-reset discipline: ≥ 2 `reset_for_slot(slot_id)` calls
        // precede the first Error event emit position (entry + exit
        // on the HB scaffold; entry + exit on hybrid Some-guard).
        // Reuse the H107 structural pin (≥ 4) and pin the source-order
        // relationship between the first Error event and the entry
        // reset block.
        let first_error_idx = fn_body
            .find("GenerationEvent::Error(")
            .expect("H109: GenerationEvent::Error emit located");
        // Count resets before the FIRST Error emit — there must be at
        // least one Error emit AFTER the first entry-reset block
        // (the iter-2-decode Error event at the prefill-Ok branch).
        // Source-order proxy: at the iter-2-decode emit position
        // (the LAST Error emit before the exit-reset block), ≥ 2
        // entry resets must precede.
        //
        // Simpler invariant: the iter-2-decode label appears AFTER
        // some `reset_for_slot(slot_id)` calls in source order.
        let iter_decode_idx = fn_body
            .find("iter-B4c-kernel-iter-2-decode")
            .expect("H109: iter-2-decode cite located (asserted above)");
        let resets_before_decode_cite = fn_body[..iter_decode_idx]
            .matches("reset_for_slot(slot_id)")
            .count();
        assert!(
            resets_before_decode_cite >= 2,
            "H109 FALSIFIED: at the source-order position of the \
             `iter-B4c-kernel-iter-2-decode` sub-deferral cite, only \
             {resets_before_decode_cite} `reset_for_slot(slot_id)` \
             calls precede it. Expected ≥ 2 (entry reset on HB \
             scaffold + entry reset on hybrid scaffold). Drift here \
             means the iter-2-decode error is emitted BEFORE the \
             entry-reset block — breaks the iter-3 entry discipline \
             pinned by H107."
        );
        // Also pin the first Error emit position is at or after some
        // entry-reset (defense against a refactor that emits Error
        // BEFORE resetting the slot).
        let _ = first_error_idx; // referenced for clarity; the
                                 // iter-2-decode cite-based pin
                                 // above is the load-bearing one.

        // The ADR-040 §6.1.35 closure block exists in the ADR.
        let adr = crate::serve::api::engine::adr040_history_doc() /* iter-230 A1: §6.1.x moved to history (aeb6e87c) */;
        assert!(
            adr.contains("### 6.1.35"),
            "H109 FALSIFIED: ADR-040 §6.1.35 closure block not found. \
             iter-3 sub-deferral cites point at a non-existent \
             destination. Update the ADR with the iter-3 closure \
             block before merging."
        );
    }
}

// ============================================================================
// ADR-040 iter-B4c-kernel iter-4 — H110-H115 hypothesis pins
// ============================================================================
//
// Scope (iter-4 advances the Gemma 4 worker-arm lift arc by one arm —
// Embed — direct mirror of Qwen35 iter-C2d-cont-kernel iter-3 §6.1.29
// for the embed surface):
//
//   * iter-1 (§6.1.31, commit `bac4c385`) shipped the Generate-arm
//     scaffold lift onto the persistent multi-seq `MultiSeqHbKvBuffers`
//     + sibling `MultiSeqHybridKvBuffers` (`reset_for_slot` primitive +
//     `generate_gemma4_once_slot_aware` orchestrator + worker-arm
//     dispatch fork).
//   * iter-2A (§6.1.32) landed the model-level slot-aware fn
//     `forward_prefill_with_soft_tokens_slot_aware` with bounds-first
//     pre-flight + 4-way KV-regime dispatch fork.
//   * iter-2B (§6.1.34, commit `1676fcd1`) landed the production-default
//     hybrid F16-K + TQ-HB-V slot routing via `MultiSeqHybridKvBuffers`
//     slice_view mount + delegate-to-sibling pattern.
//   * iter-3 (§6.1.35, commit `0c63bfe9`) shipped the GenerateStream-
//     arm slot-aware orchestrator port — `generate_stream_gemma4_once_slot_aware`
//     + worker-arm dispatch fork on both scaffolds.
//
//   * iter-4 (THIS commit, ADR-040 §6.1.36) ships:
//     - NEW `embed_gemma4_slot_aware` orchestrator at engine.rs (direct
//       mirror of Qwen35 `embed_qwen35_slot_aware` §6.1.29 + Gemma 4
//       `generate_gemma4_once_slot_aware` iter-1 + `generate_stream_gemma4_once_slot_aware`
//       iter-3 for the embed-vector result surface).
//       Reuses iter-1's `MultiSeqHbKvBuffers::reset_for_slot` + sibling
//       `MultiSeqHybridKvBuffers::reset_for_slot` primitives + iter-2A/2B's
//       `forward_prefill_with_soft_tokens_slot_aware` kernel call.
//     - `worker_run` Gemma 4 Embed-arm lift fork (take + restore on
//       `g.multi_seq_kv` + `g.multi_seq_kv_hybrid`).
//     - L2-normalized hidden-vector read from
//       `loaded.weights.activations.norm_out` (byte-equivalent to the
//       tail of `MlxModelWeights::forward_embed_last` at
//       forward_prefill.rs:2306-2331).
//     - Tests revised for the post-iter-4 lifted state: H40, H56, H62,
//       H68, H75, H108 swap their `gemma4-forward-embed-last-slot-N
//       (iter-C2c-cont` literal for the surviving SoftTokens-arm label
//       `gemma4-forward-prefill-with-soft-tokens-slot-N (iter-C2c-cont`
//       (sibling-discipline intent preserved; the C2c clamp survives
//       on the SoftTokens arm only post-iter-4).
//
// Tests (H110-H115 mirror Qwen35 iter-3 H64-H69 1:1):
//   H110 (skip-mode): SerialFifo + SlotId(0) Gemma 4 Embed dispatch
//                     byte-equivalent post-iter-4.  The `handle.slot_id
//                     != SlotId(0)` predicate short-circuits below the
//                     lift fork; the existing
//                     `g.weights.forward_embed_last(&prompt_tokens,
//                     &mut g.ctx)` dispatch at the `match &mut loaded`
//                     block fires verbatim.
//   H111 (skip-mode): iter-4 lift landed at Embed arm: the worker_run
//                     body contains a real call to
//                     `embed_gemma4_slot_aware(` under the Gemma 4
//                     Embed arm.
//   H112 (skip-mode): persistent multi_seq_kv + multi_seq_kv_hybrid
//                     take/restore at the iter-4 lift call site (both
//                     scaffolds; mirror of iter-3 take/restore pattern).
//   H113 (skip-mode): `reset_for_slot(slot_id)` at entry + exit on
//                     BOTH scaffolds (the slot-aware embed fn body has
//                     ≥ 2 occurrences for the HB scaffold AND ≥ 2 for
//                     the hybrid scaffold).
//   H114 (skip-mode): Qwen35 + Qwen3VL + Gemma 4 Generate/GenerateStream
//                     /SoftTokens worker arms unchanged by iter-4.
//                     Gemma 4 SoftTokens (iter-5) clamp still present.
//                     iter-1+2A+2B Generate-arm + iter-3 GenerateStream-
//                     arm lift fns still called (iter-4 must not
//                     regress).
//   H115 (skip-mode): embedding vector output shape preserved (return
//                     type `Result<Vec<f32>>` + L2-normalize source-
//                     order present + exit-reset AFTER prefill call).
// ----------------------------------------------------------------------------

#[cfg(test)]
mod adr040_phase_b_iter_b4c_kernel_iter4_gemma4_tests {
    // Skip-mode source-grep tests; intentionally NO `use super::*;`
    // (tests rely on `include_str!` against engine.rs + the ADR doc).

    // ── Helper: snip worker_run body the same way iter-1/2/3 tests do ──
    fn worker_run_body(src: &str) -> &str {
        let body_start = src
            .find("fn worker_run(")
            .expect("iter-4: worker_run entry not found");
        let body_after = &src[body_start..];
        let body_end_off = body_after
            .find("\n// The worker thread for `LoadedModel::Qwen35` returns a sentinel error")
            .or_else(|| body_after.find("\n/// Worker-thread entry point"))
            .unwrap_or(body_after.len().min(200_000));
        &body_after[..body_end_off]
    }

    /// **H110 (skip-mode)** — SerialFifo + SlotId(0) AND SlotAware +
    /// SlotId(0) Gemma 4 Embed dispatch is byte-equivalent post-iter-4.
    ///
    /// Source-grep pin: the iter-4 lift fork at the Embed arm uses the
    /// predicate `handle.slot_id != SlotId(0)`.  SerialFifo always
    /// hands out SlotId(0) (FifoSchedulerAdapter invariant); SlotAware's
    /// first request also gets SlotId(0).  In both cases the predicate
    /// is FALSE → the lift block falls through to the existing
    /// `match &mut loaded { LoadedModel::Gemma(g) =>
    ///     g.weights.forward_embed_last(&prompt_tokens, &mut g.ctx) }`
    /// dispatch, byte-equivalent to pre-iter-4 + pre-C2c.
    ///
    /// Defends the H1 / H2 / H23 / H41 / H44 / H77 / H104 byte-
    /// equivalence chain extended to the Gemma 4 embed surface.  Direct
    /// mirror of Qwen35 iter-C2d-cont-kernel iter-3 H64.
    #[test]
    fn h110_slot_id_0_gemma4_embed_routes_through_forward_embed_last_byte_equivalent() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        // The pre-iter-4 `forward_embed_last` dispatch must still be
        // reachable from the worker arm for the SlotId(0) fallback.
        assert!(
            body.contains("g.weights.forward_embed_last(&prompt_tokens, &mut g.ctx)"),
            "H110 FALSIFIED: post-iter-4 worker_run Gemma 4 Embed \
             dispatch no longer routes through \
             `g.weights.forward_embed_last(&prompt_tokens, &mut g.ctx)` \
             for SlotId(0).  The iter-4 lift fork must be ADDITIVE \
             (sibling above the `match &mut loaded` dispatch), NOT \
             REPLACE the SerialFifo / SlotId(0) path.  SerialFifo + \
             SlotId(0) byte-equivalence (H1 / H2 / H77 / H104 chain) \
             is BROKEN for the Gemma 4 embed arm."
        );

        // The lift fork predicate at the Gemma 4 worker arms must be
        // `matches!(loaded, LoadedModel::Gemma(_)) && handle.slot_id != SlotId(0)`
        // — the same shape iter-1 / iter-3 used.  Pin: at least FOUR
        // occurrences of the literal predicate in the worker_run body
        // (iter-1 Generate lift + iter-3 GenerateStream lift + iter-4
        // Embed lift + iter-5 SoftTokens clamp).
        let predicate = "matches!(loaded, LoadedModel::Gemma(_)) && handle.slot_id != SlotId(0)";
        let n = body.matches(predicate).count();
        assert!(
            n >= 4,
            "H110 FALSIFIED: the iter-4 lift fork predicate \
             `{predicate}` must appear at least 4 times in worker_run \
             body (one for iter-1 Generate lift, one for iter-3 \
             GenerateStream lift, one for iter-4 Embed lift, one for \
             iter-5 SoftTokens clamp).  Got {n}.  Drift here means the \
             lift may fire at SlotId(0) too, breaking byte-equivalence."
        );
    }

    /// **H111 (skip-mode)** — iter-4 Embed-arm lift landed at
    /// `worker_run`: the slot-aware fn `embed_gemma4_slot_aware` is
    /// called from the worker_run body at the Gemma 4 Embed arm.
    /// Source-grep pin.  Mirror of Qwen35 iter-3 H65.
    #[test]
    fn h111_iter4_lift_landed_for_gemma4_embed_arm() {
        let src = include_str!("engine.rs");
        // The slot-aware fn is defined in this file.
        assert!(
            src.contains("fn embed_gemma4_slot_aware("),
            "H111 FALSIFIED: `embed_gemma4_slot_aware` is NOT defined \
             in engine.rs.  iter-B4c-kernel iter-4 production surface \
             MISSING — orchestrator not landed."
        );
        let body = worker_run_body(src);
        assert!(
            body.contains("embed_gemma4_slot_aware("),
            "H111 FALSIFIED: worker_run does NOT call \
             `embed_gemma4_slot_aware`.  iter-4 lift is not wired into \
             the dispatch fork — the Embed-arm SlotId(N>0) routing is \
             missing."
        );

        // The slot-aware fn passes `slot_id` (the SlotId from the
        // admit'd handle), NOT a hard-coded SlotId(0).  Pin via
        // substring search inside the lift call block.
        let lift_block_start = body
            .find("embed_gemma4_slot_aware(")
            .expect("H111: lift call site not found");
        let lift_block_end = body[lift_block_start..]
            .find(");")
            .map(|off| lift_block_start + off + 2)
            .unwrap_or(body.len().min(lift_block_start + 2000));
        let lift_block = &body[lift_block_start..lift_block_end];
        assert!(
            lift_block.contains("slot_id"),
            "H111 FALSIFIED: the lift call site does not pass `slot_id` \
             into `embed_gemma4_slot_aware`.  The iter-4 lift must \
             thread the admit'd SlotHandle's slot_id into the slot-\
             aware fn.  Got block: {lift_block}"
        );
        assert!(
            !lift_block.contains(", SlotId(0),"),
            "H111 FALSIFIED: lift call site contains a hard-coded \
             `SlotId(0)` literal argument.  Per-slot routing is broken \
             — the orchestrator must receive the admit'd handle's \
             SlotId verbatim."
        );
    }

    /// **H112 (skip-mode)** — persistent multi_seq_kv +
    /// multi_seq_kv_hybrid take+restore pattern at the iter-4 lift
    /// call site.  Mirror of Qwen35 iter-3 H66 with both scaffolds.
    ///
    /// Pin both `g.multi_seq_kv.take()` + `g.multi_seq_kv_hybrid.take()`
    /// AND the corresponding restores after the call.  The take+restore
    /// pattern is required for:
    /// (a) three-iter symmetry — iter-1 already established this pattern
    ///     for the Generate arm with BOTH scaffolds (iter-2B); iter-3
    ///     extended it to the GenerateStream arm; iter-4 must use the
    ///     same shape so the persistent-scaffold invariants hold
    ///     across BOTH Generate + GenerateStream + Embed requests at
    ///     any slot.
    /// (b) defense against the same regressions H79 / H99 / H106 catch
    ///     — forgotten put-back → next request finds `is_none()`
    ///     defense-in-depth typed error; clone-instead-of-take →
    ///     persistent scaffold's per-slot state is not actually
    ///     mutated, defeating cross-request isolation.
    ///
    /// iter-4's take+restore is ADDITIVE — the worker_run body has
    /// iter-1+2B Generate + iter-3 GenerateStream + iter-4 Embed take+
    /// restores.  Pin via count ≥ 3 for both take and restore on the
    /// HB scaffold, and ≥ 3 on the hybrid scaffold.
    #[test]
    fn h112_lift_call_site_takes_and_restores_both_scaffolds() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        // HB scaffold take pattern (mirror of iter-1 H79 + iter-3 H106).
        let hb_take_count = body.matches("g.multi_seq_kv.take()").count();
        assert!(
            hb_take_count >= 3,
            "H112 FALSIFIED: the iter-4 lift call site does not \
             `take()` the persistent HB scaffold out of \
             `GemmaLoadedModel.multi_seq_kv`.  Expected at least 3 \
             occurrences of `g.multi_seq_kv.take()` in worker_run body \
             (one each for iter-1 Generate + iter-3 GenerateStream + \
             iter-4 Embed lift forks); got {hb_take_count}."
        );
        // HB scaffold restore pattern.
        let hb_restore_count = body.matches("g.multi_seq_kv = Some(multi_seq)").count();
        assert!(
            hb_restore_count >= 3,
            "H112 FALSIFIED: the iter-4 lift call site does not put \
             the persistent HB scaffold back into `g.multi_seq_kv` \
             after the slot-aware embed fn returns.  Expected at least \
             3 occurrences of `g.multi_seq_kv = Some(multi_seq)` in \
             worker_run body (one each for iter-1 + iter-3 + iter-4 \
             lift forks); got {hb_restore_count}.  The next request to \
             land at SlotId(N>0) would find `multi_seq_kv.is_none()` \
             and hit the defense-in-depth typed error."
        );
        // Hybrid scaffold take pattern (mirror of iter-2B H99 + iter-3
        // H106).
        let hyb_take_count = body.matches("g.multi_seq_kv_hybrid.take()").count();
        assert!(
            hyb_take_count >= 3,
            "H112 FALSIFIED: the iter-4 lift call site does not \
             `take()` the persistent hybrid scaffold out of \
             `GemmaLoadedModel.multi_seq_kv_hybrid`.  Expected at least \
             3 occurrences of `g.multi_seq_kv_hybrid.take()` in \
             worker_run body (one each for iter-2B Generate + iter-3 \
             GenerateStream + iter-4 Embed lift forks); got \
             {hyb_take_count}.  Production-default hybrid path \
             (HF2Q_HYBRID_KV=1 per H10) would silently surface the \
             iter-2A hybrid typed error or route to slot 0's region."
        );
        // Hybrid scaffold restore pattern.
        let hyb_restore_count = body
            .matches("g.multi_seq_kv_hybrid = multi_seq_hybrid")
            .count();
        assert!(
            hyb_restore_count >= 3,
            "H112 FALSIFIED: the iter-4 lift call site does not put \
             the persistent hybrid scaffold back into \
             `g.multi_seq_kv_hybrid` after the slot-aware embed fn \
             returns.  Expected at least 3 occurrences of \
             `g.multi_seq_kv_hybrid = multi_seq_hybrid` in worker_run \
             body (one each for iter-2B + iter-3 + iter-4 lift forks); \
             got {hyb_restore_count}."
        );
    }

    /// **H113 (skip-mode)** — per-slot reset at entry + exit of the
    /// slot-aware embed fn on BOTH scaffolds (HB + hybrid).  Mirror of
    /// Qwen35 iter-3 H67 with the dual-scaffold discipline iter-1
    /// established for Gemma 4.
    ///
    /// Pin: ≥ 4 occurrences of `reset_for_slot(slot_id)` inside the
    /// slot-aware embed fn body (entry + exit on HB scaffold via
    /// `multi_seq_kv.iter_mut()` + entry + exit on hybrid scaffold
    /// via `if let Some(ref mut hybrid_scaffold) = multi_seq_kv_hybrid`
    /// Option guards).
    #[test]
    fn h113_slot_aware_embed_fn_calls_reset_for_slot_at_entry_and_exit() {
        // reset_for_slot primitives (iter-1's load-bearing add).
        let kv_src = include_str!("../../../src/inference/models/gemma4/kv_cache.rs");
        assert!(
            kv_src.contains("pub fn reset_for_slot("),
            "H113 FALSIFIED: `reset_for_slot` is not defined in \
             gemma4/kv_cache.rs.  iter-4 inherits this primitive from \
             iter-1; if it's gone, iter-1 was reverted."
        );

        let src = include_str!("engine.rs");
        let fn_marker = "fn embed_gemma4_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H113: embed_gemma4_slot_aware not defined");
        // Window covering the fn body — bound by next top-level fn
        // marker or end-of-file.
        let body_after = &src[fn_idx..];
        let body_end_off = body_after[fn_marker.len()..]
            .find("\nfn ")
            .map(|off| off + fn_marker.len())
            .unwrap_or(body_after.len().min(60_000));
        let fn_body = &body_after[..body_end_off];

        // Total reset_for_slot count ≥ 4: entry + exit on HB scaffold
        // (2) + entry + exit on hybrid scaffold (2 — wrapped in
        // Option Some-guards).
        let reset_calls = fn_body.matches("reset_for_slot(slot_id)").count();
        assert!(
            reset_calls >= 4,
            "H113 FALSIFIED: `embed_gemma4_slot_aware` must call \
             `reset_for_slot(slot_id)` at LEAST 4 times (entry + exit \
             on the HB scaffold via `multi_seq_kv.iter_mut()` + entry \
             + exit on the hybrid scaffold via `if let Some(ref mut \
             hybrid_scaffold) = multi_seq_kv_hybrid` Option guards).  \
             Got {reset_calls} call(s).  Drift here means cross-request \
             isolation is BROKEN on at least one scaffold."
        );

        // Per-layer iteration on HB scaffold.
        assert!(
            fn_body.contains("multi_seq_kv.iter_mut()"),
            "H113 FALSIFIED: slot-aware embed fn does NOT iterate \
             per-layer via `multi_seq_kv.iter_mut()`.  Per-layer reset \
             coverage on HB scaffold is incomplete."
        );
        // Per-layer iteration on hybrid scaffold (inside Some-guard).
        assert!(
            fn_body.contains("hybrid_scaffold.iter_mut()"),
            "H113 FALSIFIED: slot-aware embed fn does NOT iterate \
             per-layer via `hybrid_scaffold.iter_mut()` inside the \
             hybrid Option Some-guard.  Per-layer reset coverage on \
             the production-default hybrid scaffold is incomplete."
        );
    }

    /// **H114 (skip-mode)** — Qwen35 + Qwen3VL + Gemma 4
    /// Generate/GenerateStream/SoftTokens worker arms unchanged by
    /// iter-4.  Mirror of H82 / H108 extended for the iter-4 narrowing:
    /// iter-4 replaces the iter-4 Gemma 4 Embed clamp, so only the
    /// iter-5 (GenerateWithSoftTokens) clamp remains in the Gemma 4
    /// worker_run surface (Generate + GenerateStream are also lifted
    /// post-iter-1+2A+2B + iter-3).
    #[test]
    fn h114_other_worker_arms_unchanged_by_iter4() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        // Qwen35 lift fns from iter-C2d-cont-kernel iter-1/2/3/4 all
        // STILL called.
        for lift_fn in [
            "super::engine_qwen35::generate_qwen35_once_slot_aware(",
            "super::engine_qwen35::generate_stream_qwen35_once_extended_slot_aware(",
            "super::engine_qwen35::embed_qwen35_slot_aware(",
            "super::engine_qwen35::generate_qwen35_once_with_soft_tokens_slot_aware(",
        ] {
            assert!(
                body.contains(lift_fn),
                "H114 FALSIFIED: Qwen35 lift fn `{lift_fn}` is NOT \
                 called from worker_run.  iter-4 must NOT regress any \
                 Qwen35 worker-arm lift (§6.1.27/28/29/30)."
            );
        }

        // iter-1 Gemma 4 Generate-arm lift fn STILL called (iter-4
        // must not regress iter-1).
        assert!(
            body.contains("generate_gemma4_once_slot_aware("),
            "H114 FALSIFIED: iter-1 Gemma 4 Generate lift fn \
             `generate_gemma4_once_slot_aware` is NOT called from \
             worker_run.  iter-4 must NOT regress iter-1's Generate-arm \
             lift (§6.1.31)."
        );

        // iter-3 Gemma 4 GenerateStream-arm lift fn STILL called
        // (iter-4 must not regress iter-3).
        assert!(
            body.contains("generate_stream_gemma4_once_slot_aware("),
            "H114 FALSIFIED: iter-3 Gemma 4 GenerateStream lift fn \
             `generate_stream_gemma4_once_slot_aware` is NOT called \
             from worker_run.  iter-4 must NOT regress iter-3's \
             GenerateStream-arm lift (§6.1.35)."
        );

        // ADR-040 iter-C2e (2026-05-30 §6.1.52) — Qwen3VL clamp
        // SHIPPED post-Gemma 4 iter-4. Sibling discipline pin:
        // Gemma 4 iter-4 must not REMOVE the C2e Qwen3VL clamp.
        assert!(
            body.contains(
                "matches!(loaded, LoadedModel::Qwen3VlText(_)) && handle.slot_id != SlotId(0)"
            ),
            "H114 FALSIFIED (post-C2e revision per §6.1.52): Qwen3VL \
             clamp missing from worker_run.  iter-C2e SHIPPED 2026-05-30 \
             adds the Qwen3VL clamp at the four worker arms; Gemma 4 \
             iter-4 must NOT regress the C2e clamp."
        );

        // ADR-040 iter-B4c-kernel iter-5 (§6.1.37 — TERMINAL Gemma 4
        // worker-arm lift) REVISES H114: the SoftTokens clamp label
        // `gemma4-forward-prefill-with-soft-tokens-slot-N (iter-C2c-cont`
        // is LEGITIMATELY REMOVED by iter-5.  Sibling-discipline intent
        // ("iter-4 must NOT touch the SoftTokens arm") preserved via
        // the positive assertion that the iter-5 lift fn is called
        // from worker_run (iter-4 did not author this lift; iter-5
        // did — but iter-4 must NOT regress it).
        assert!(
            body.contains("generate_gemma4_once_with_soft_tokens_slot_aware("),
            "H114 FALSIFIED (post-iter-5 revision per §6.1.37): \
             Gemma 4 iter-5 TERMINAL SoftTokens lift fn \
             `generate_gemma4_once_with_soft_tokens_slot_aware` is \
             NOT called from worker_run.  iter-4 must NOT regress \
             iter-5's §6.1.37 lift."
        );
        assert!(
            body.contains("iter-B4c-kernel-iter-5"),
            "H114 FALSIFIED: iter-5 sub-deferral cite \
             `iter-B4c-kernel-iter-5` missing from worker_run.  \
             SoftTokens-arm clamp's typed-deferral discipline broken."
        );

        // iter-4 lifts the Gemma 4 Embed arm — the Embed clamp label
        // (`gemma4-forward-embed-last-slot-N (iter-C2c-cont`) is
        // legitimately REMOVED from worker_run.  Pin its absence as
        // structural witness of the lift.
        assert!(
            !body.contains("gemma4-forward-embed-last-slot-N (iter-C2c-cont"),
            "H114 FALSIFIED: Gemma 4 Embed arm clamp label \
             `gemma4-forward-embed-last-slot-N (iter-C2c-cont` is \
             STILL present in worker_run.  iter-4 must REPLACE the \
             Embed-arm typed clamp with the actual `embed_gemma4_slot_aware` \
             lift call — the surviving label indicates the lift was \
             not actually applied (the iter-1 §6.1.31 / iter-3 §6.1.35 \
             relabeled cite would survive)."
        );
    }

    /// **H115 (skip-mode)** — embedding vector output shape preserved
    /// by the slot-aware embed fn.  Source-grep + structural pin on
    /// engine.rs for the body of `embed_gemma4_slot_aware`:
    /// (a) the fn signature returns `Result<Vec<f32>>` (NOT
    ///     `Result<GenerationResult>` or `Result<u32>`);
    /// (b) the fn body calls `forward_prefill_with_soft_tokens_slot_aware(`
    ///     (the iter-2A/2B slot-aware kernel call that lands the
    ///     prefill bytes into the slot's region of the persistent KV);
    /// (c) the fn body contains the L2-normalize idiom (the `/=` denom
    ///     + the 1e-12 epsilon floor — byte-equivalent to the tail of
    ///     `MlxModelWeights::forward_embed_last`);
    /// (d) the exit-reset call runs AFTER the prefill call (so the
    ///     embed result is not accidentally truncated by the reset;
    ///     the reset is per-slot KV state, not per-fn output).
    ///
    /// This pin defends against three regression classes: (a) iter-4
    /// returning a `GenerationResult` (decode-shaped surface) which
    /// would break the embed-as-vector contract handlers depend on,
    /// (b) iter-4 inverting the reset/prefill order (resetting AFTER
    /// the forward but discarding the output, or resetting BEFORE
    /// entry and BEFORE forward only — both break the per-slot
    /// isolation invariant), and (c) iter-4 dropping the L2 normalize
    /// (would break cosine-similarity-by-dot-product downstream).
    #[test]
    fn h115_slot_aware_embed_fn_preserves_embedding_vector_shape() {
        let src = include_str!("engine.rs");
        let fn_marker = "fn embed_gemma4_slot_aware(";
        let fn_start = src
            .find(fn_marker)
            .expect("H115: embed_gemma4_slot_aware not defined");
        // Locate the fn body — bound by next top-level `fn ` or
        // end-of-file.
        let body_after = &src[fn_start..];
        let body_end_off = body_after[fn_marker.len()..]
            .find("\nfn ")
            .map(|off| off + fn_marker.len())
            .unwrap_or(body_after.len().min(60_000));
        let fn_body = &body_after[..body_end_off];

        // (a) Return type is `Result<Vec<f32>>` — distinguishes embed
        // from generate (which returns `Result<GenerationResult>`).
        assert!(
            fn_body.contains("-> Result<Vec<f32>>"),
            "H115 FALSIFIED: `embed_gemma4_slot_aware` return type is \
             not `Result<Vec<f32>>`.  The embed surface returns the L2-\
             normalized hidden vector (length `hidden_size`); a \
             different return type breaks the embed-as-vector contract \
             handlers depend on.  Mirror of `forward_embed_last` shape."
        );

        // (b) The fn body calls `forward_prefill_with_soft_tokens_slot_aware`
        // — the iter-2A/2B slot-aware kernel call.
        assert!(
            fn_body.contains("forward_prefill_with_soft_tokens_slot_aware("),
            "H115 FALSIFIED: `embed_gemma4_slot_aware` does not call \
             `forward_prefill_with_soft_tokens_slot_aware(`.  The \
             embed surface must route through the iter-2A/2B slot-\
             aware prefill kernel; calling a different forward fn \
             would break the slot-isolation invariant + the embed-as-\
             vector byte-equivalence baseline."
        );

        // (c) L2-normalize idiom present (the denom + epsilon floor +
        // in-place `/=` per element — byte-equivalent to
        // forward_embed_last:2326-2330).
        assert!(
            fn_body.contains("1e-12"),
            "H115 FALSIFIED: `embed_gemma4_slot_aware` body does not \
             contain the `1e-12` L2-normalize epsilon floor (matches \
             the BERT-lane `bert_l2_normalize_gpu` epsilon).  Drift \
             here means consumers cannot compute cosine similarity by \
             dot product — breaks the embed contract."
        );
        assert!(
            fn_body.contains("*v /= denom"),
            "H115 FALSIFIED: `embed_gemma4_slot_aware` body does not \
             contain the in-place `*v /= denom` L2-normalize step.  \
             Drift here means the output vector is not normalized."
        );

        // (d) The exit-reset call runs AFTER the prefill call.
        // Source-order pin: the LAST `reset_for_slot(slot_id)` in the
        // body must appear AFTER `forward_prefill_with_soft_tokens_slot_aware`.
        // Otherwise the exit-reset is misplaced.
        let last_reset = fn_body
            .rfind("reset_for_slot(slot_id)")
            .expect("H115: at least one reset_for_slot(slot_id) call expected");
        let prefill_pos = fn_body
            .find("forward_prefill_with_soft_tokens_slot_aware(")
            .expect("H115: forward_prefill_with_soft_tokens_slot_aware call site expected");
        assert!(
            last_reset > prefill_pos,
            "H115 FALSIFIED: the LAST `reset_for_slot(slot_id)` call \
             (source-order position {last_reset}) appears BEFORE the \
             `forward_prefill_with_soft_tokens_slot_aware(` call \
             (source-order position {prefill_pos}).  The exit-reset \
             MUST run AFTER the prefill so the per-slot cleanup \
             happens on the way out (mirrors iter-1 / iter-3 exit-reset \
             discipline).  Inverting the order breaks per-slot \
             isolation for the next request."
        );

        // The ADR-040 §6.1.36 closure block exists in the ADR.
        let adr = crate::serve::api::engine::adr040_history_doc() /* iter-230 A1: §6.1.x moved to history (aeb6e87c) */;
        assert!(
            adr.contains("### 6.1.36"),
            "H115 FALSIFIED: ADR-040 §6.1.36 closure block not found. \
             iter-4 sub-deferral cites point at a non-existent \
             destination. Update the ADR with the iter-4 closure \
             block before merging."
        );
    }
}

/// **ADR-040 Phase B iter-B4c-kernel iter-5 (TERMINAL Gemma 4 worker-arm
/// lift, 2026-05-30)** — H116-H122 hypothesis pins for the Gemma 4
/// GenerateWithSoftTokens-arm slot-aware orchestrator port + the
/// vision-augmented streaming branch lift in
/// `generate_stream_gemma4_once_slot_aware`.
///
/// **Direct mirror of Qwen35 iter-C2d-cont-kernel iter-4 §6.1.30** for
/// the Gemma 4 architecture's vision-aware soft-token surface.  Same
/// hypothesis shape as the iter-1 (H77-H83), iter-3 (H104-H109), and
/// iter-4 (H110-H115) test modules: worker_run lift-fork predicate pin
/// + slot-aware fn body pin + persistent scaffold take+restore pin
/// + per-slot reset pin + sibling-discipline (other arms unchanged)
/// pin + sub-deferrals coverage pin + TERMINAL pin (no surviving Gemma
/// 4 worker-arm clamp remains).
#[cfg(test)]
#[allow(non_snake_case, clippy::too_many_arguments)]
mod adr040_phase_b_iter_b4c_kernel_iter5_gemma4_tests {
    // Skip-mode source-grep tests; intentionally NO `use super::*;`
    // (tests rely on `include_str!` against engine.rs + the ADR doc).

    // ── Helper: snip worker_run body the same way iter-1/3/4 tests do ──
    fn worker_run_body(src: &str) -> &str {
        let body_start = src
            .find("fn worker_run(")
            .expect("iter-5: worker_run entry not found");
        let body_after = &src[body_start..];
        let body_end_off = body_after
            .find("\n// The worker thread for `LoadedModel::Qwen35` returns a sentinel error")
            .or_else(|| body_after.find("\n/// Worker-thread entry point"))
            .unwrap_or(body_after.len().min(200_000));
        &body_after[..body_end_off]
    }

    /// **H116 (skip-mode)** — SerialFifo + SlotId(0) AND SlotAware +
    /// SlotId(0) Gemma 4 GenerateWithSoftTokens dispatch is byte-
    /// equivalent post-iter-5.
    ///
    /// Source-grep pin: the iter-5 lift fork at the GenerateWithSoftTokens
    /// arm uses the predicate `handle.slot_id != SlotId(0)`.  SerialFifo
    /// always hands out SlotId(0) (FifoSchedulerAdapter invariant);
    /// SlotAware's first request also gets SlotId(0).  In both cases the
    /// predicate is FALSE → the lift block falls through to the existing
    /// `match &mut loaded { LoadedModel::Gemma(g) =>
    ///     generate_once_with_soft_tokens(g, ..) }` dispatch, byte-
    /// equivalent to pre-iter-5 + pre-C2c.
    ///
    /// Defends the H1 / H2 / H23 / H41 / H44 / H77 / H104 / H110 byte-
    /// equivalence chain extended to the Gemma 4 vision-aware soft-token
    /// surface.  Direct mirror of Qwen35 iter-C2d-cont-kernel iter-4 H70.
    #[test]
    fn h116_slot_id_0_gemma4_soft_tokens_routes_through_generate_once_with_soft_tokens_byte_equivalent(
    ) {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        // The pre-iter-5 `generate_once_with_soft_tokens` dispatch must
        // still be reachable from the worker arm for the SlotId(0)
        // fallback.
        assert!(
            body.contains("generate_once_with_soft_tokens("),
            "H116 FALSIFIED: post-iter-5 worker_run Gemma 4 \
             GenerateWithSoftTokens dispatch no longer routes through \
             `generate_once_with_soft_tokens` for SlotId(0).  The iter-5 \
             lift fork must be ADDITIVE (sibling above the `match &mut \
             loaded` dispatch), NOT REPLACE the SerialFifo / SlotId(0) \
             path.  SerialFifo + SlotId(0) byte-equivalence \
             (H1 / H2 / H77 / H104 / H110 chain) is BROKEN for the Gemma \
             4 vision-aware soft-token arm."
        );

        // The lift fork predicate at the Gemma 4 worker arms must be
        // `matches!(loaded, LoadedModel::Gemma(_)) && handle.slot_id != SlotId(0)`
        // — the same shape iter-1 / iter-3 / iter-4 used.  Pin: at least
        // FOUR occurrences of the literal predicate in the worker_run
        // body (iter-1 Generate lift + iter-3 GenerateStream lift +
        // iter-4 Embed lift + iter-5 SoftTokens lift).
        let predicate = "matches!(loaded, LoadedModel::Gemma(_)) && handle.slot_id != SlotId(0)";
        let n = body.matches(predicate).count();
        assert!(
            n >= 4,
            "H116 FALSIFIED: the iter-5 lift fork predicate \
             `{predicate}` must appear at least 4 times in worker_run \
             body (iter-1 Generate + iter-3 GenerateStream + iter-4 \
             Embed + iter-5 SoftTokens lifts).  Got {n}.  Drift here \
             means the lift may fire at SlotId(0) too, breaking byte-\
             equivalence."
        );
    }

    /// **H117 (skip-mode)** — iter-5 SoftTokens-arm lift landed at
    /// `worker_run`: the slot-aware fn
    /// `generate_gemma4_once_with_soft_tokens_slot_aware` is called
    /// from the worker_run body at the Gemma 4 GenerateWithSoftTokens
    /// arm.  Source-grep pin.  Mirror of Qwen35 iter-4 H71.
    #[test]
    fn h117_iter5_lift_landed_for_gemma4_soft_tokens_arm() {
        let src = include_str!("engine.rs");
        // The slot-aware fn is defined in this file.
        assert!(
            src.contains("fn generate_gemma4_once_with_soft_tokens_slot_aware("),
            "H117 FALSIFIED: `generate_gemma4_once_with_soft_tokens_slot_aware` \
             is NOT defined in engine.rs.  iter-B4c-kernel iter-5 production \
             surface MISSING — orchestrator not landed."
        );
        let body = worker_run_body(src);
        assert!(
            body.contains("generate_gemma4_once_with_soft_tokens_slot_aware("),
            "H117 FALSIFIED: worker_run does NOT call \
             `generate_gemma4_once_with_soft_tokens_slot_aware`.  iter-5 \
             lift is not wired into the dispatch fork — the \
             GenerateWithSoftTokens-arm SlotId(N>0) routing is missing."
        );

        // The slot-aware fn passes `slot_id` (the SlotId from the
        // admit'd handle), NOT a hard-coded SlotId(0).  Pin via
        // substring search inside the lift call block.
        let lift_block_start = body
            .find("generate_gemma4_once_with_soft_tokens_slot_aware(")
            .expect("H117: lift call site not found");
        let lift_block_end = body[lift_block_start..]
            .find(");")
            .map(|off| lift_block_start + off + 2)
            .unwrap_or(body.len().min(lift_block_start + 2000));
        let lift_block = &body[lift_block_start..lift_block_end];
        assert!(
            lift_block.contains("slot_id"),
            "H117 FALSIFIED: the lift call site does not pass `slot_id` \
             into `generate_gemma4_once_with_soft_tokens_slot_aware`.  \
             The iter-5 lift must thread the admit'd SlotHandle's \
             slot_id into the slot-aware fn.  Got block: {lift_block}"
        );
        assert!(
            !lift_block.contains(", SlotId(0),"),
            "H117 FALSIFIED: lift call site contains a hard-coded \
             `SlotId(0)` literal argument.  Per-slot routing is broken \
             — the orchestrator must receive the admit'd handle's \
             SlotId verbatim."
        );

        // The iter-5 typed-clamp label `gemma4-forward-prefill-with-soft-tokens-slot-N`
        // is REMOVED from worker_run (iter-5 legitimately removes it
        // by lifting).
        assert!(
            !body.contains("gemma4-forward-prefill-with-soft-tokens-slot-N (iter-C2c-cont"),
            "H117 FALSIFIED: Gemma 4 GenerateWithSoftTokens arm clamp \
             label `gemma4-forward-prefill-with-soft-tokens-slot-N \
             (iter-C2c-cont` is STILL present in worker_run.  iter-5 \
             must REPLACE the SoftTokens-arm typed clamp with the actual \
             `generate_gemma4_once_with_soft_tokens_slot_aware` lift \
             call — the surviving label indicates the lift was not \
             actually applied."
        );
    }

    /// **H118 (skip-mode)** — persistent both-scaffolds take/restore
    /// at the iter-5 lift call site.  Mirror of Qwen35 iter-4 H72 +
    /// Gemma 4 iter-4 H112 with both scaffolds.
    ///
    /// Pin both `g.multi_seq_kv.take()` + `g.multi_seq_kv_hybrid.take()`
    /// AND the corresponding restores after the call.  The take+restore
    /// pattern is required for:
    /// (a) four-iter symmetry — iter-1 established the pattern for the
    ///     Generate arm with BOTH scaffolds (iter-2B); iter-3 extended
    ///     to GenerateStream; iter-4 extended to Embed; iter-5 must
    ///     use the same shape so the persistent-scaffold invariants
    ///     hold across all FOUR Gemma 4 worker arms.
    /// (b) defense against the regressions H79 / H99 / H106 / H112
    ///     catch — forgotten put-back → next request finds `is_none()`
    ///     defense-in-depth typed error.
    ///
    /// iter-5's take+restore is ADDITIVE — the worker_run body has
    /// iter-1+2B Generate + iter-3 GenerateStream + iter-4 Embed +
    /// iter-5 SoftTokens take+restores.  Pin via count ≥ 4 for both
    /// take and restore on the HB scaffold, and ≥ 4 on the hybrid
    /// scaffold.
    #[test]
    fn h118_persistent_both_scaffolds_take_restore_at_iter5_call_site() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        // HB scaffold take pattern (mirror of iter-1 H79 + iter-3 H106
        // + iter-4 H112).
        let hb_take_count = body.matches("g.multi_seq_kv.take()").count();
        assert!(
            hb_take_count >= 4,
            "H118 FALSIFIED: the iter-5 lift call site does not \
             `take()` the persistent HB scaffold out of \
             `GemmaLoadedModel.multi_seq_kv`.  Expected at least 4 \
             occurrences of `g.multi_seq_kv.take()` in worker_run body \
             (iter-1 Generate + iter-3 GenerateStream + iter-4 Embed + \
             iter-5 SoftTokens lift forks); got {hb_take_count}."
        );
        // HB scaffold restore pattern.
        let hb_restore_count = body.matches("g.multi_seq_kv = Some(multi_seq)").count();
        assert!(
            hb_restore_count >= 4,
            "H118 FALSIFIED: the iter-5 lift call site does not put \
             the persistent HB scaffold back into `g.multi_seq_kv` \
             after the slot-aware soft-tokens fn returns.  Expected at \
             least 4 occurrences of `g.multi_seq_kv = Some(multi_seq)` \
             in worker_run body (iter-1 + iter-3 + iter-4 + iter-5 \
             lift forks); got {hb_restore_count}.  The next request to \
             land at SlotId(N>0) would find `multi_seq_kv.is_none()` \
             and hit the defense-in-depth typed error."
        );
        // Hybrid scaffold take pattern.
        let hyb_take_count = body.matches("g.multi_seq_kv_hybrid.take()").count();
        assert!(
            hyb_take_count >= 4,
            "H118 FALSIFIED: the iter-5 lift call site does not \
             `take()` the persistent hybrid scaffold out of \
             `GemmaLoadedModel.multi_seq_kv_hybrid`.  Expected at least \
             4 occurrences of `g.multi_seq_kv_hybrid.take()` in \
             worker_run body (iter-2B Generate + iter-3 GenerateStream \
             + iter-4 Embed + iter-5 SoftTokens lift forks); got \
             {hyb_take_count}.  Production-default hybrid path \
             (HF2Q_HYBRID_KV=1 per H10) would silently surface the \
             iter-2A hybrid typed error or route to slot 0's region."
        );
        // Hybrid scaffold restore pattern.
        let hyb_restore_count = body
            .matches("g.multi_seq_kv_hybrid = multi_seq_hybrid")
            .count();
        assert!(
            hyb_restore_count >= 4,
            "H118 FALSIFIED: the iter-5 lift call site does not put \
             the persistent hybrid scaffold back into \
             `g.multi_seq_kv_hybrid` after the slot-aware soft-tokens \
             fn returns.  Expected at least 4 occurrences of \
             `g.multi_seq_kv_hybrid = multi_seq_hybrid` in worker_run \
             body (iter-2B + iter-3 + iter-4 + iter-5 lift forks); got \
             {hyb_restore_count}."
        );
    }

    /// **H119 (skip-mode)** — per-slot reset at entry + exit of the
    /// slot-aware soft-tokens fn on BOTH scaffolds (HB + hybrid).
    /// Mirror of Qwen35 iter-4 H73 + Gemma 4 iter-4 H113 with the
    /// dual-scaffold discipline iter-1 established for Gemma 4.
    ///
    /// Pin: ≥ 4 occurrences of `reset_for_slot(slot_id)` inside the
    /// slot-aware soft-tokens fn body (entry + exit on HB scaffold via
    /// `multi_seq_kv.iter_mut()` + entry + exit on hybrid scaffold via
    /// `if let Some(ref mut hybrid_scaffold) = multi_seq_kv_hybrid`
    /// Option guards).
    #[test]
    fn h119_slot_aware_soft_tokens_fn_calls_reset_for_slot_at_entry_and_exit() {
        // reset_for_slot primitives (iter-1's load-bearing add).
        let kv_src = include_str!("../../../src/inference/models/gemma4/kv_cache.rs");
        assert!(
            kv_src.contains("pub fn reset_for_slot("),
            "H119 FALSIFIED: `reset_for_slot` is not defined in \
             gemma4/kv_cache.rs.  iter-5 inherits this primitive from \
             iter-1; if it's gone, iter-1 was reverted."
        );

        let src = include_str!("engine.rs");
        let fn_marker = "fn generate_gemma4_once_with_soft_tokens_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H119: generate_gemma4_once_with_soft_tokens_slot_aware not defined");
        // Window covering the fn body — bound by next top-level fn
        // marker or end-of-file.
        let body_after = &src[fn_idx..];
        let body_end_off = body_after[fn_marker.len()..]
            .find("\nfn ")
            .map(|off| off + fn_marker.len())
            .unwrap_or(body_after.len().min(60_000));
        let fn_body = &body_after[..body_end_off];

        // Total reset_for_slot count ≥ 4: entry + exit on HB scaffold
        // (2) + entry + exit on hybrid scaffold (2 — wrapped in Option
        // Some-guards).
        let reset_calls = fn_body.matches("reset_for_slot(slot_id)").count();
        assert!(
            reset_calls >= 4,
            "H119 FALSIFIED: `generate_gemma4_once_with_soft_tokens_slot_aware` \
             must call `reset_for_slot(slot_id)` at LEAST 4 times (entry \
             + exit on the HB scaffold via `multi_seq_kv.iter_mut()` + \
             entry + exit on the hybrid scaffold via `if let Some(ref \
             mut hybrid_scaffold) = multi_seq_kv_hybrid` Option guards). \
             Got {reset_calls} call(s).  Drift here means cross-request \
             isolation is BROKEN on at least one scaffold."
        );

        // Per-layer iteration on HB scaffold.
        assert!(
            fn_body.contains("multi_seq_kv.iter_mut()"),
            "H119 FALSIFIED: slot-aware soft-tokens fn does NOT iterate \
             per-layer via `multi_seq_kv.iter_mut()`.  Per-layer reset \
             coverage on HB scaffold is incomplete."
        );
        // Per-layer iteration on hybrid scaffold (inside Some-guard).
        assert!(
            fn_body.contains("hybrid_scaffold.iter_mut()"),
            "H119 FALSIFIED: slot-aware soft-tokens fn does NOT iterate \
             per-layer via `hybrid_scaffold.iter_mut()` inside the \
             hybrid Option Some-guard.  Per-layer reset coverage on \
             the production-default hybrid scaffold is incomplete."
        );

        // Forward call site: the slot-aware soft-tokens fn must thread
        // the caller's `soft_tokens` slice through to the slot-aware
        // prefill kernel — NOT pass `&[]` (which would be the Generate-
        // arm shape).  Pin: the prefill call site contains the literal
        // `soft_tokens,` argument (positional in the call).
        let prefill_pos = fn_body
            .find("forward_prefill_with_soft_tokens_slot_aware(")
            .expect("H119: forward_prefill_with_soft_tokens_slot_aware call site expected");
        let prefill_block_end = fn_body[prefill_pos..]
            .find(");")
            .map(|off| prefill_pos + off + 2)
            .unwrap_or(fn_body.len().min(prefill_pos + 3000));
        let prefill_block = &fn_body[prefill_pos..prefill_block_end];
        assert!(
            prefill_block.contains("soft_tokens"),
            "H119 FALSIFIED: slot-aware soft-tokens prefill call site \
             does not thread `soft_tokens` into the kernel call.  The \
             iter-5 lift must carry the caller's vision-aware soft-token \
             overrides through to the kernel.  Got block: {prefill_block}"
        );
        // And it must NOT pass `&[]` as the soft_tokens argument
        // (that would be the Generate-arm shape — iter-1+2A+2B uses
        // `&[]`; iter-5 must NOT).
        assert!(
            !prefill_block.contains("&[], // SoftTokens"),
            "H119 FALSIFIED: slot-aware soft-tokens prefill call site \
             passes `&[]` as the soft_tokens argument.  The iter-5 lift \
             must thread the caller's `soft_tokens` slice verbatim."
        );
    }

    /// **H120 (skip-mode)** — vision-augmented streaming at SlotId(N>0)
    /// LIFTED.  Mirror of Qwen35 iter-4 H74 for the Gemma 4 streaming
    /// surface.
    ///
    /// Pre-iter-5: `generate_stream_gemma4_once_slot_aware` (iter-3
    /// §6.1.35) surfaced a typed SSE Error event when `soft_tokens` was
    /// non-empty, citing iter-B4c-kernel-iter-5 as the deferred surface.
    /// Post-iter-5: that abort path is REMOVED — the soft_tokens slice
    /// is threaded verbatim through to the slot-aware prefill kernel.
    /// Pin (a) the iter-3 abort path's typed-error substring is REMOVED
    /// from the stream-arm fn body; pin (b) the prefill call site now
    /// passes `soft_tokens` (not `&[]`); pin (c) the iter-3 abort path's
    /// guard `if !soft_tokens.is_empty()` is also REMOVED.
    #[test]
    fn h120_vision_augmented_streaming_slot_n_gt_0_lifted() {
        let src = include_str!("engine.rs");
        let fn_marker = "fn generate_stream_gemma4_once_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H120: generate_stream_gemma4_once_slot_aware not defined");
        let body_after = &src[fn_idx..];
        let body_end_off = body_after[fn_marker.len()..]
            .find("\nfn ")
            .map(|off| off + fn_marker.len())
            .unwrap_or(body_after.len().min(60_000));
        let fn_body = &body_after[..body_end_off];

        // (a) The iter-3 typed-error substring "vision-augmented
        // streaming slot-aware port is iter-B4c-kernel-iter-5" is
        // REMOVED from the streaming fn body (iter-5 legitimately
        // removes it by lifting).
        assert!(
            !fn_body.contains("vision-augmented streaming slot-aware port is"),
            "H120 FALSIFIED: the iter-3 typed-error substring \
             `vision-augmented streaming slot-aware port is` is STILL \
             present in `generate_stream_gemma4_once_slot_aware`'s \
             body.  iter-5 must REMOVE the iter-3 abort path — \
             vision-augmented streaming at SlotId(N>0) now routes \
             through the kernel verbatim."
        );

        // (b) The prefill call site now passes `soft_tokens` (not
        // `&[]`).  Locate the prefill call site + look for the
        // `soft_tokens,` positional argument.
        let prefill_pos = fn_body
            .find("forward_prefill_with_soft_tokens_slot_aware(")
            .expect("H120: forward_prefill_with_soft_tokens_slot_aware call site expected");
        let prefill_block_end = fn_body[prefill_pos..]
            .find(");")
            .map(|off| prefill_pos + off + 2)
            .unwrap_or(fn_body.len().min(prefill_pos + 3000));
        let prefill_block = &fn_body[prefill_pos..prefill_block_end];
        assert!(
            prefill_block.contains("soft_tokens"),
            "H120 FALSIFIED: streaming-arm prefill call site does not \
             thread `soft_tokens` into the kernel call.  The iter-5 \
             lift must carry the caller's vision-aware soft-token \
             overrides through to the kernel even on the streaming \
             surface.  Got block: {prefill_block}"
        );

        // (c) iter-5 cite present in the fn body (the comment narrating
        // the lift).
        assert!(
            fn_body.contains("iter-B4c-kernel iter-5"),
            "H120 FALSIFIED: streaming-arm fn body does not cite \
             `iter-B4c-kernel iter-5` — the lift narration is missing, \
             which would make it harder for future iters to grep the \
             lift site."
        );
    }

    /// **H121 (skip-mode)** — Qwen35 + Qwen3VL worker arms unchanged
    /// by iter-5.  Mirror of H82 / H108 / H114 extended for the iter-5
    /// narrowing: iter-5 replaces the LAST Gemma 4 worker-arm clamp
    /// (GenerateWithSoftTokens), so all FOUR Gemma 4 lifts are now
    /// engaged.
    #[test]
    fn h121_qwen35_qwen3vl_and_other_gemma4_arms_unchanged_by_iter5() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        // Qwen35 lift fns from iter-C2d-cont-kernel iter-1/2/3/4 all
        // STILL called.
        for lift_fn in [
            "super::engine_qwen35::generate_qwen35_once_slot_aware(",
            "super::engine_qwen35::generate_stream_qwen35_once_extended_slot_aware(",
            "super::engine_qwen35::embed_qwen35_slot_aware(",
            "super::engine_qwen35::generate_qwen35_once_with_soft_tokens_slot_aware(",
        ] {
            assert!(
                body.contains(lift_fn),
                "H121 FALSIFIED: Qwen35 lift fn `{lift_fn}` is NOT \
                 called from worker_run.  iter-5 must NOT regress any \
                 Qwen35 worker-arm lift (§6.1.27/28/29/30)."
            );
        }

        // iter-1 Gemma 4 Generate-arm lift fn STILL called (iter-5
        // must not regress iter-1).
        assert!(
            body.contains("generate_gemma4_once_slot_aware("),
            "H121 FALSIFIED: iter-1 Gemma 4 Generate lift fn \
             `generate_gemma4_once_slot_aware` is NOT called from \
             worker_run.  iter-5 must NOT regress iter-1's Generate-arm \
             lift (§6.1.31)."
        );

        // iter-3 Gemma 4 GenerateStream-arm lift fn STILL called.
        assert!(
            body.contains("generate_stream_gemma4_once_slot_aware("),
            "H121 FALSIFIED: iter-3 Gemma 4 GenerateStream lift fn \
             `generate_stream_gemma4_once_slot_aware` is NOT called \
             from worker_run.  iter-5 must NOT regress iter-3's \
             GenerateStream-arm lift (§6.1.35)."
        );

        // iter-4 Gemma 4 Embed-arm lift fn STILL called.
        assert!(
            body.contains("embed_gemma4_slot_aware("),
            "H121 FALSIFIED: iter-4 Gemma 4 Embed lift fn \
             `embed_gemma4_slot_aware` is NOT called from worker_run.  \
             iter-5 must NOT regress iter-4's Embed-arm lift (§6.1.36)."
        );

        // ADR-040 iter-C2e (2026-05-30 §6.1.52) — Qwen3VL clamp
        // SHIPPED post-Gemma 4 iter-5. Sibling discipline pin:
        // Gemma 4 iter-5 must not REMOVE the C2e Qwen3VL clamp.
        assert!(
            body.contains(
                "matches!(loaded, LoadedModel::Qwen3VlText(_)) && handle.slot_id != SlotId(0)"
            ),
            "H121 FALSIFIED (post-C2e revision per §6.1.52): Qwen3VL \
             clamp missing from worker_run.  iter-C2e SHIPPED 2026-05-30 \
             adds the Qwen3VL clamp at the four worker arms; Gemma 4 \
             iter-5 must NOT regress the C2e clamp."
        );
    }

    /// **H122 (skip-mode) — TERMINAL Gemma 4 worker-arm lift pin.**
    /// Mirror of Qwen35 iter-4 H76 for the Gemma 4 architecture.
    ///
    /// Post-iter-5: NONE of `iter-B4c-kernel-iter-{1,2A,2B,3,4,5}`
    /// appear as a typed-clamp label pattern (e.g.
    /// `-slot-N (iter-C2c-cont per ADR-040 §6.1.21 / iter-B4c-kernel`)
    /// in worker_run.  iter-1/2A/2B/3/4/5 lift fns all wired (covered
    /// by H114 / H121 above).  ADR-040 §6.1.37 closure block exists +
    /// names `iter-B4c-kernel iter-5` + marks TERMINAL.  Surviving sub-
    /// deferrals (iter-2A-cont, iter-2B-xlen, iter-2C, iter-2D, iter-
    /// 2-decode, iter-LCP, iter-G) are orthogonal kernel-side refactors,
    /// NOT arm lifts.
    #[test]
    fn h122_terminal_gemma4_worker_arm_lift_pin() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);

        // No surviving Gemma 4 worker-arm clamp labels (all four
        // arm-lift clamps now legitimately removed by iter-1/3/4/5).
        for arm_clamp_label in [
            "gemma4-forward-prefill-slot-N (iter-C2c-cont",
            "gemma4-forward-embed-last-slot-N (iter-C2c-cont",
            "gemma4-forward-prefill-with-soft-tokens-slot-N (iter-C2c-cont",
        ] {
            assert!(
                !body.contains(arm_clamp_label),
                "H122 FALSIFIED: Gemma 4 worker-arm clamp label \
                 `{arm_clamp_label}` is STILL present in worker_run.  \
                 iter-5 is TERMINAL — every Gemma 4 worker-arm-lift \
                 clamp must be REMOVED by iter-1/3/4/5.  Surviving \
                 label indicates the lift was not actually applied."
            );
        }

        // The ADR-040 §6.1.37 closure block exists in the ADR.
        let adr = crate::serve::api::engine::adr040_history_doc() /* iter-230 A1: §6.1.x moved to history (aeb6e87c) */;
        assert!(
            adr.contains("### 6.1.37"),
            "H122 FALSIFIED: ADR-040 §6.1.37 closure block not found. \
             iter-5 sub-deferral cites point at a non-existent \
             destination. Update the ADR with the iter-5 closure block \
             before merging."
        );
        // The §6.1.37 block names `iter-B4c-kernel iter-5`.
        let block_start = adr.find("### 6.1.37").expect("H122: §6.1.37 block missing");
        let block_end_off = adr[block_start..]
            .find("\n### ")
            .map(|off| block_start + off)
            .unwrap_or(adr.len());
        let block = &adr[block_start..block_end_off];
        assert!(
            block.contains("iter-B4c-kernel iter-5"),
            "H122 FALSIFIED: ADR-040 §6.1.37 closure block does NOT \
             name `iter-B4c-kernel iter-5`.  The closure block must \
             explicitly identify the iter shipped."
        );
        // The §6.1.37 block marks TERMINAL.
        assert!(
            block.contains("TERMINAL"),
            "H122 FALSIFIED: ADR-040 §6.1.37 closure block does NOT \
             mark iter-5 as TERMINAL.  The Gemma 4 worker-arm lift arc \
             is complete; the closure block must say so so future iters \
             can grep the terminal pin."
        );

        // Surviving sub-deferrals named in §6.1.37 — orthogonal kernel
        // refactors / opt-in surfaces.  Pin at least the load-bearing
        // sub-iters by exact substring (operator-grep'able).
        for sub_def in [
            "iter-B4c-kernel-iter-2-decode",
            "iter-B4c-kernel-iter-2A-cont",
            "iter-B4c-kernel-iter-2B-xlen",
        ] {
            assert!(
                block.contains(sub_def),
                "H122 FALSIFIED: ADR-040 §6.1.37 closure block does \
                 NOT name surviving sub-deferral `{sub_def}`.  Every \
                 surviving deferral must have an operator-grep'able \
                 iter-N label."
            );
        }
    }
}

// ============================================================================
// ADR-040 iter-B4c-kernel iter-2-decode-A — H123-H129 hypothesis pins
// ============================================================================
//
// Scope (iter-2-decode-A 2026-05-30 — Gemma 4 multi-token decode-loop body
// wrapping forward_decode at slot_id):
//
// Pre-iter-2-decode-A, the 3 Gemma 4 slot-aware orchestrators
// (generate_gemma4_once_slot_aware / generate_stream_gemma4_once_slot_aware
// / generate_gemma4_once_with_soft_tokens_slot_aware) each surfaced a typed
// `MultiSeqError::CapabilityUnsupported { capability: "...iter-B4c-kernel-
// iter-2-decode per ADR-040 §6.1.{32,35,37}..." }` after the iter-2B
// prefill returned its first decode token — the multi-token decode-loop
// body wrapping `forward_decode` at slot_id was the named sub-deferral.
//
// iter-2-decode-A SHIPS:
//   * NEW `MlxModelWeights::forward_decode_slot_aware` in
//     `src/serve/forward_prefill.rs` (~430 LOC body) — mirror of
//     `forward_prefill_with_soft_tokens_slot_aware`'s slot-view mount +
//     delegate-to-sibling pattern applied to the decode body.  Bounds-
//     first preflight + 4-way KV-regime dispatch fork (HF2Q_USE_DENSE /
//     cb_bits==0 / HF2Q_HYBRID_KV / HB-encoded default).  Production-
//     default hybrid F16-K + TQ-HB-V branch ships REAL slot routing via
//     slice_view mount on `self.hybrid_kv` → delegate to the unchanged
//     sibling `forward_decode` at `gemma4/forward_gpu.rs:310` → restore
//     on exit.
//   * REPLACED Generate-arm IIFE typed-error body in
//     `generate_gemma4_once_slot_aware` with a real greedy decode loop
//     calling `forward_decode_slot_aware` per token.  EOS / max_tokens
//     handling + tokenizer fragment accumulation matching the
//     `generate_once` greedy fast-path shape at engine.rs:7728-7800.
//     iter-2-decode-C sampling-clamp at the loop entry: any request
//     with T>0 / grammar / stop_strings / logprobs surfaces typed
//     CapabilityUnsupported naming iter-2-decode-C.
//   * REPLACED GenerateStream-arm IIFE typed-error body in
//     `generate_stream_gemma4_once_slot_aware` with a real per-token
//     Delta-emission decode loop + terminal Done event.  Mirror of the
//     Generate-arm landing but routed through the SSE channel.
//   * REPLACED SoftTokens-arm IIFE typed-error body in
//     `generate_gemma4_once_with_soft_tokens_slot_aware` with the same
//     greedy decode loop (the SoftTokens vs Generate difference is fully
//     consumed by the prefill call's soft_tokens param; the decode body
//     is identical to Generate-arm).
//
// Sub-deferrals (typed CapabilityUnsupported labels):
//   * iter-B4c-kernel-iter-2-decode-B: HB-encoded HF2Q_HYBRID_KV=0 opt-out
//     decode-side slot routing (mirror of iter-2A-cont prefill scope).
//     Surfaced from the new fn body's HB-encoded branch.
//   * iter-B4c-kernel-iter-2-decode-C: orchestrator-side full sampler /
//     grammar / tool-call / stop-strings / logprobs / reasoning-text
//     surface.  Surfaced from the 3 orchestrator decode-loop heads.
//   * iter-B4c-kernel-iter-2-decode-D: dense F32 (HF2Q_USE_DENSE=1) +
//     legacy 4-bit (HF2Q_TQ_CODEBOOK_BITS=4) decode-side slot routing.
//     Surfaced from the new fn body's dense / legacy branches.
//   * iter-B4c-kernel-iter-2-decode-A-xlen: BF16 xlen K/V decode-side
//     slot routing (HF2Q_DFLASH_XLEN_SDPA=1 opt-in).
//
// Tests (H123-H129):
//   H123 (skip-mode): forward_decode_slot_aware signature lands with
//                     slot_id: SlotId + multi_seq_kv_hb + multi_seq_kv_hybrid
//                     params; sibling forward_decode signature UNCHANGED.
//   H124 (skip-mode): new fn body uses slice_view + mount on self.hybrid_kv
//                     + delegate to forward_decode + restore — mirror of
//                     iter-2B prefill pattern.
//   H125 (skip-mode): Generate orchestrator IIFE typed-error body REPLACED
//                     with real decode loop calling forward_decode_slot_aware.
//   H126 (skip-mode): GenerateStream orchestrator IIFE typed-error body
//                     REPLACED with real Delta-emission decode loop +
//                     terminal Done event.
//   H127 (skip-mode): SoftTokens orchestrator IIFE typed-error body
//                     REPLACED with real decode loop.
//   H128 (skip-mode): SerialFifo byte-equivalence preserved — the sibling
//                     forward_decode signature contains NO slot_id /
//                     multi_seq_kv params.  Code-path disjointness via
//                     the worker-arm SlotId(0) predicate.
//   H129 (skip-mode): Qwen35 + Qwen3VL + Embed-arm (iter-4) UNCHANGED;
//                     iter-1/2A/2B/3/5 lift scaffolds PRESERVED.

#[cfg(test)]
mod adr040_phase_b_iter_b4c_kernel_iter2_decode_a_gemma4_tests {
    // Skip-mode source-grep tests; intentionally NO `use super::*;`.

    /// **H123 (skip-mode)** — New fn `forward_decode_slot_aware` IS defined
    /// on `MlxModelWeights` in `src/serve/forward_prefill.rs` with the
    /// `slot_id: SlotId` + `multi_seq_kv_hb` + `multi_seq_kv_hybrid` params.
    /// Returns `Result<u32>` matching the sibling fn's return shape.
    ///
    /// Pin defends regression to the iter-5 IIFE pattern where the
    /// orchestrator's hypothetical-Ok branch returned typed
    /// `CapabilityUnsupported` instead of calling a real model-fn.
    #[test]
    fn h123_new_fn_forward_decode_slot_aware_landed() {
        let src = include_str!("../forward_prefill.rs");
        let fn_marker = "pub fn forward_decode_slot_aware(";
        let fn_idx = src.find(fn_marker).expect(
            "H123 FALSIFIED: `forward_decode_slot_aware` is NOT defined in \
             src/serve/forward_prefill.rs. iter-2-decode-A's load-bearing \
             model-fn primitive is missing — orchestrator decode loops have \
             nothing to call.",
        );
        let sig_window = &src[fn_idx..(fn_idx + 2500).min(src.len())];
        // Required params (mirror of iter-2A/2B prefill signature).
        for required in [
            "input_token: u32",
            "seq_pos: usize",
            "gpu: &mut GpuContext",
            "slot_id: SlotId",
            "multi_seq_kv_hb: &mut Vec<MultiSeqHbKvBuffers>",
            "multi_seq_kv_hybrid: Option<&mut Vec<MultiSeqHybridKvBuffers>>",
        ] {
            assert!(
                sig_window.contains(required),
                "H123 FALSIFIED: new fn signature missing `{required}`. \
                 iter-2-decode-A's load-bearing param surface broken."
            );
        }
        // Return type: Result<u32> (matches sibling forward_decode).
        assert!(
            sig_window.contains(") -> Result<u32>"),
            "H123 FALSIFIED: new fn does NOT return `Result<u32>`. \
             Decode-loop callers expect the on-GPU greedy argmax."
        );
    }

    /// **H124 (skip-mode)** — New fn body uses the iter-2B slice_view
    /// mount + delegate-to-sibling pattern for the HYBRID branch
    /// (production-default per H10 falsification at §6.1.11).
    ///
    /// Required structural elements:
    ///   * `.slice_view(` — per-slot view primitive (Qwen35 B4a-cont mirror).
    ///   * `self.hybrid_kv = Some(slot_view_hybrid)` — mount.
    ///   * `self.forward_decode(` — delegate to the unchanged sibling.
    ///   * `self.hybrid_kv = prior_hybrid_kv` — restore on exit.
    ///
    /// Pin defends regression where iter-2-decode-A accidentally drops one
    /// of the 4 load-bearing structural elements (mount without restore,
    /// delegate without mount, etc.).
    #[test]
    fn h124_new_fn_slice_view_mount_delegate_pattern() {
        let src = include_str!("../forward_prefill.rs");
        let fn_marker = "pub fn forward_decode_slot_aware(";
        let fn_idx = src.find(fn_marker).expect("H124: fn marker present (H123)");
        // Look at the next ~30k bytes to cover the full fn body.
        // ADR-040 iter-2C + iter-2D (§6.1.46) — window bumped from
        // 30_000 to 80_000 to cover both the hybrid + HB-encoded
        // branches now that the dense F32 + legacy 4-bit branches
        // sit before them.
        let fn_window = &src[fn_idx..(fn_idx + 80_000).min(src.len())];
        for required in [
            ".slice_view(",
            "self.hybrid_kv = Some(",
            // ADR-040 S1c-2: the delegate is now the capture-parameterized
            // `forward_decode_impl` (forward_decode_slot_aware{,_capture_hidden}
            // are thin wrappers passing capture_hidden=false/true).
            "self.forward_decode_impl(",
            "self.hybrid_kv = prior_hybrid_kv",
        ] {
            assert!(
                fn_window.contains(required),
                "H124 FALSIFIED: new fn body missing structural element \
                 `{required}`. iter-2-decode-A's slot-view mount + delegate \
                 + restore pattern broken — per-slot routing through the \
                 persistent multi-seq scaffold cannot work."
            );
        }
    }

    /// **H125 (skip-mode)** — Generate orchestrator IIFE typed-error body
    /// REPLACED with a real decode loop calling forward_decode_slot_aware.
    ///
    /// (a) The OLD iter-2-decode label (`gemma4-forward-prefill-kernel-slot-N-
    ///     decode-loop (iter-B4c-kernel-iter-2-decode per ADR-040 §6.1.32`)
    ///     is REMOVED from the Generate orchestrator body.
    /// (b) The new fn call (`forward_decode_slot_aware(`) IS present at
    ///     least once in the Generate orchestrator body.
    /// (c) The orchestrator threads `slot_id` (not a hardcoded SlotId(0))
    ///     into the new fn call.
    #[test]
    fn h125_generate_orchestrator_decode_loop_wired() {
        let src = include_str!("engine.rs");
        let fn_marker = "fn generate_gemma4_once_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H125: generate_gemma4_once_slot_aware not found");
        // Cover the full fn body — generously sized window.
        let fn_window = &src[fn_idx..(fn_idx + 20_000).min(src.len())];
        // (a) OLD iter-2-decode literal REMOVED.
        let old_label =
            "gemma4-forward-prefill-kernel-slot-N-decode-loop (iter-B4c-kernel-iter-2-decode per ADR-040 §6.1.32";
        assert!(
            !fn_window.contains(old_label),
            "H125 FALSIFIED: Generate orchestrator body still contains the \
             iter-2-decode IIFE typed-error label `{old_label}`. iter-2-decode-A \
             did not actually wire the decode loop — orchestrator still \
             returns CapabilityUnsupported after the prefill Ok."
        );
        // (b) New fn call present.
        assert!(
            fn_window.contains(".forward_decode_slot_aware("),
            "H125 FALSIFIED: Generate orchestrator body does NOT call \
             `forward_decode_slot_aware`. iter-2-decode-A decode loop \
             is missing — orchestrator cannot emit content tokens."
        );
        // (c) slot_id threaded through (not a hardcoded SlotId(0)).
        let call_idx = fn_window
            .find(".forward_decode_slot_aware(")
            .expect("H125: call marker present (asserted above)");
        let call_window = &fn_window[call_idx..(call_idx + 800).min(fn_window.len())];
        assert!(
            call_window.contains("slot_id"),
            "H125 FALSIFIED: Generate orchestrator call site does NOT pass \
             `slot_id`.  Per-slot decode routing broken."
        );
        assert!(
            !call_window.contains(", SlotId(0),"),
            "H125 FALSIFIED: Generate orchestrator call site contains a \
             literal `SlotId(0)` argument.  Per-slot decode routing broken."
        );
    }

    /// **H126 (skip-mode)** — GenerateStream orchestrator IIFE typed-error
    /// body REPLACED with a real Delta-emission decode loop + Done event.
    ///
    /// (a) The OLD iter-2-decode stream label (`gemma4-forward-prefill-
    ///     kernel-slot-N-stream-decode-loop (iter-B4c-kernel-iter-2-decode
    ///     per ADR-040 §6.1.35`) is REMOVED from the stream orchestrator.
    /// (b) The new fn call (`forward_decode_slot_aware(`) IS present.
    /// (c) The stream emits Delta events (the per-token content fragments)
    ///     AND a terminal Done event.
    #[test]
    fn h126_generate_stream_orchestrator_decode_loop_wired() {
        let src = include_str!("engine.rs");
        let fn_marker = "fn generate_stream_gemma4_once_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H126: generate_stream_gemma4_once_slot_aware not found");
        // Generous body window — streaming fn is larger than the sync arm.
        // iter-2-decode-C inflates the streaming arm body to ~29k bytes
        // (full sampler/grammar/stop-strings/logprobs/reasoning surface);
        // window widened to 50k to cover the trailing Done event.
        let fn_window = &src[fn_idx..(fn_idx + 50_000).min(src.len())];
        // (a) OLD streaming iter-2-decode literal REMOVED.
        let old_label =
            "gemma4-forward-prefill-kernel-slot-N-stream-decode-loop (iter-B4c-kernel-iter-2-decode per ADR-040 §6.1.35";
        assert!(
            !fn_window.contains(old_label),
            "H126 FALSIFIED: GenerateStream orchestrator body still contains \
             the iter-2-decode stream IIFE typed-error label `{old_label}`. \
             iter-2-decode-A did not actually wire the stream decode loop."
        );
        // (b) New fn call present.
        assert!(
            fn_window.contains(".forward_decode_slot_aware("),
            "H126 FALSIFIED: GenerateStream orchestrator body does NOT call \
             `forward_decode_slot_aware`. iter-2-decode-A stream decode loop \
             is missing."
        );
        // (c) Delta event emit + Done event emit present in the body.
        assert!(
            fn_window.contains("GenerationEvent::Delta {"),
            "H126 FALSIFIED: GenerateStream orchestrator body does NOT emit \
             `GenerationEvent::Delta {{` events.  The per-token Content delta \
             emission for the SSE stream is missing — clients would receive \
             no decoded content."
        );
        assert!(
            fn_window.contains("GenerationEvent::Done {"),
            "H126 FALSIFIED: GenerateStream orchestrator body does NOT emit \
             a terminal `GenerationEvent::Done {{` event.  SSE stream cannot \
             terminate cleanly — clients hang."
        );
    }

    /// **H127 (skip-mode)** — SoftTokens orchestrator IIFE typed-error body
    /// REPLACED with a real decode loop calling forward_decode_slot_aware.
    ///
    /// Mirror of H125 for the SoftTokens-arm.  The SoftTokens-arm difference
    /// is fully consumed by the prefill call's `soft_tokens` parameter; the
    /// decode body should be identical to Generate-arm.
    #[test]
    fn h127_soft_tokens_orchestrator_decode_loop_wired() {
        let src = include_str!("engine.rs");
        let fn_marker = "fn generate_gemma4_once_with_soft_tokens_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H127: generate_gemma4_once_with_soft_tokens_slot_aware not found");
        let fn_window = &src[fn_idx..(fn_idx + 20_000).min(src.len())];
        // (a) OLD iter-2-decode soft-tokens literal REMOVED.
        let old_label =
            "gemma4-forward-prefill-kernel-slot-N-soft-tokens-decode-loop (iter-B4c-kernel-iter-2-decode per ADR-040 §6.1.37";
        assert!(
            !fn_window.contains(old_label),
            "H127 FALSIFIED: SoftTokens orchestrator body still contains the \
             iter-2-decode soft-tokens IIFE typed-error label `{old_label}`. \
             iter-2-decode-A did not actually wire the SoftTokens decode loop."
        );
        // (b) New fn call present.
        assert!(
            fn_window.contains(".forward_decode_slot_aware("),
            "H127 FALSIFIED: SoftTokens orchestrator body does NOT call \
             `forward_decode_slot_aware`. iter-2-decode-A SoftTokens decode \
             loop is missing — vision-aware chat completion cannot emit \
             content tokens at SlotId(N>0)."
        );
        // (c) slot_id threaded through.
        let call_idx = fn_window
            .find(".forward_decode_slot_aware(")
            .expect("H127: call marker present (asserted above)");
        let call_window = &fn_window[call_idx..(call_idx + 800).min(fn_window.len())];
        assert!(
            call_window.contains("slot_id"),
            "H127 FALSIFIED: SoftTokens orchestrator call site does NOT pass \
             `slot_id`.  Per-slot decode routing broken."
        );
    }

    /// **H128 (skip-mode)** — SerialFifo + SlotId(0) byte-equivalence
    /// preserved at the sibling-fn signature level.  The sibling
    /// `forward_decode` in `gemma4/forward_gpu.rs:310` MUST NOT contain
    /// `slot_id` or `multi_seq_kv*` in its signature.
    ///
    /// Pin defends H1/H2/H23/H41/H44/H77/H102 byte-equivalence chain at
    /// the decode-side model-fn signature level — code-path disjointness
    /// is the load-bearing invariant.
    #[test]
    fn h128_serial_fifo_sibling_forward_decode_signature_unchanged() {
        let src = include_str!("../../inference/models/gemma4/forward_gpu.rs");
        let sibling_marker = "pub fn forward_decode(";
        let sib_idx = src
            .find(sibling_marker)
            .expect("H128: sibling forward_decode signature missing");
        // Look at the signature only (NOT the body — body may legitimately
        // reference slot_id via doc-comments narrating iter-2-decode-A).
        let sig_end = src[sib_idx..]
            .find(") -> Result<u32>")
            .map(|off| sib_idx + off + ") -> Result<u32>".len())
            .unwrap_or(sib_idx + 600);
        let sig_window = &src[sib_idx..sig_end.min(src.len())];
        assert!(
            !sig_window.contains("slot_id"),
            "H128 FALSIFIED: sibling `forward_decode` signature contains \
             `slot_id` parameter. iter-2-decode-A discipline broken — the \
             sibling fn MUST remain byte-equivalent for SerialFifo + \
             SlotId(0).  iter-2-decode-A's primitive is a NEW sibling fn \
             (`forward_decode_slot_aware` in forward_prefill.rs); the \
             existing sibling MUST NOT be touched."
        );
        assert!(
            !sig_window.contains("multi_seq_kv"),
            "H128 FALSIFIED: sibling `forward_decode` signature mentions \
             `multi_seq_kv`. iter-2-decode-A discipline broken — SerialFifo \
             decode path MUST NOT consume the multi-seq scaffold."
        );
    }

    /// **H129 (skip-mode)** — Qwen35 + Qwen3VL + Embed-arm UNCHANGED.
    /// The iter-1/2A/2B/3/5 lift scaffolds (Generate / GenerateStream /
    /// SoftTokens orchestrators) are PRESERVED — iter-2-decode-A is
    /// PURELY ADDITIVE to those scaffolds (the decode-loop wiring lands
    /// inside the same orchestrator bodies the prior iters established).
    #[test]
    fn h129_orthogonal_surfaces_unchanged() {
        let src = include_str!("engine.rs");
        // The Qwen35 lift fns must still be called from worker_run.
        for qwen35_fn in [
            "generate_qwen35_once_slot_aware(",
            "generate_stream_qwen35_once_extended_slot_aware(",
            "embed_qwen35_slot_aware(",
            "generate_qwen35_once_with_soft_tokens_slot_aware(",
        ] {
            assert!(
                src.contains(qwen35_fn),
                "H129 FALSIFIED: Qwen35 slot-aware fn call `{qwen35_fn}` is \
                 NOT present in engine.rs. iter-2-decode-A accidentally \
                 removed a Qwen35 lift — the Qwen35 worker-arm arc (TERMINAL \
                 post-iter-C2d-cont-kernel-iter-4 §6.1.30) is COMPLETE and \
                 MUST NOT be regressed."
            );
        }
        // The Gemma 4 iter-1/2A/2B/3/4/5 lift fns must still be defined +
        // called.  iter-2-decode-A is additive INSIDE these fns; the fn
        // definitions + worker_run call sites MUST be preserved.
        for gemma_fn in [
            "fn generate_gemma4_once_slot_aware(",
            "fn generate_stream_gemma4_once_slot_aware(",
            "fn embed_gemma4_slot_aware(",
            "fn generate_gemma4_once_with_soft_tokens_slot_aware(",
        ] {
            assert!(
                src.contains(gemma_fn),
                "H129 FALSIFIED: Gemma 4 iter-1/3/4/5 lift fn `{gemma_fn}` is \
                 NOT defined. iter-2-decode-A accidentally regressed a prior \
                 iter's lift surface — every Gemma 4 worker-arm arc fn MUST \
                 be preserved."
            );
        }
        // The Embed-arm has NO decode loop (iter-4 §6.1.36 closure: "no
        // decode loop, so the iter-B4c-kernel-iter-2-decode sub-deferral
        // does NOT apply").  Defense-in-depth: the Embed-arm fn body must
        // NOT call forward_decode_slot_aware (decode loop would corrupt
        // the L2-normalized embedding vector by overwriting norm_out).
        let embed_marker = "fn embed_gemma4_slot_aware(";
        let embed_idx = src
            .find(embed_marker)
            .expect("H129: embed_gemma4_slot_aware not found");
        let embed_window = &src[embed_idx..(embed_idx + 10_000).min(src.len())];
        assert!(
            !embed_window.contains(".forward_decode_slot_aware("),
            "H129 FALSIFIED: Embed-arm fn body calls \
             `forward_decode_slot_aware`. The Embed-arm has NO decode loop \
             — calling forward_decode_slot_aware would corrupt the \
             L2-normalized embedding vector at norm_out."
        );
        // ADR-040 §6.1.38 closure block exists (forward-pin destination).
        let adr = crate::serve::api::engine::adr040_history_doc() /* iter-230 A1: §6.1.x moved to history (aeb6e87c) */;
        assert!(
            adr.contains("### 6.1.38"),
            "H129 FALSIFIED: ADR-040 §6.1.38 closure block not found. \
             iter-2-decode-A's sub-deferral cites point at a non-existent \
             destination."
        );
    }
}

// ───────────────────────────────────────────────────────────────────
// ADR-040 Phase B iter-B4c-kernel iter-2-decode-C — Gemma 4
// orchestrator-side FULL sampler / grammar / stop-strings / logprobs /
// reasoning-text surface at SlotId(N>0).
// ───────────────────────────────────────────────────────────────────
//
// iter-2-decode-A (§6.1.38) landed the production-default greedy
// fast-path: at SlotId(N>0) for hybrid F16-K + TQ-HB-V (default since
// ADR-029 iter-13), the 3 Gemma 4 worker-arm orchestrators (Generate /
// GenerateStream / SoftTokens) call the new fn
// `forward_decode_slot_aware` per token until EOS / max_tokens.  The
// sampling clamp at each orchestrator's loop entry surfaced typed
// `MultiSeqError::CapabilityUnsupported` naming
// `iter-B4c-kernel-iter-2-decode-C` for any request that engaged
// `temperature > 0.0 || grammar.is_some() || !stop_strings.is_empty()
// || logprobs`.
//
// iter-2-decode-C (this iter) REPLACES those 3 sampling clamps with the
// REAL surface mirrored from the non-slot-aware sibling `generate_once`
// slow path at engine.rs:7427-7866 and `generate_stream_once` at
// engine.rs:11008+.  The structurally-honest scope decision:
//
//   * **Generate-arm (non-streaming)**: full surface — temperature /
//     top_p / top_k / repetition_penalty / logit_bias sampling via
//     `sampler_pure::sample_token` + per-token logprobs via
//     `sample_token_with_logprob`; grammar mask + accept_bytes per
//     step + grammar-dead termination; stop_strings detection +
//     trailing strip; reasoning-text split via `split_full_output` at
//     end-of-decode.  NO surviving sub-deferral for the
//     non-streaming Generate-arm.
//
//   * **GenerateStream-arm (streaming)**: full sampler / grammar /
//     stop_strings / logprobs surface via SSE Delta + Logprobs events.
//     **Sub-deferral: streaming tool-call body emission via
//     `ToolCallStreamEmitter`** (Wave 3 W-B3 incremental-arguments
//     emission, ~200 LOC of stateful JSON parsing) — typed
//     `CapabilityUnsupported` naming
//     `iter-B4c-kernel-iter-2-decode-C-stream-tool-call per
//     ADR-040 §6.1.39`.  Requests that engage a `ToolCallSplitter`
//     are deferred; pure sampling / grammar / stop_strings / logprobs
//     / reasoning-text streaming requests proceed end-to-end.
//
//   * **SoftTokens-arm (vision-aware)**: full surface identical to
//     Generate-arm — the SoftTokens-vs-Generate difference is fully
//     consumed by the prefill call's `soft_tokens` parameter; the
//     decode body's sampler / grammar / stop-string / logprobs /
//     reasoning-text shape is identical.  NO surviving sub-deferral.
//
// iter-2-decode-C SHIPS:
//   * REPLACED `generate_gemma4_once_slot_aware`'s iter-2-decode-C
//     sampling-clamp with the FULL non-streaming sampler/grammar/
//     stop-strings/logprobs/reasoning-text surface.  Lifts the
//     `_registration` param to `registration` so the reasoning
//     splitter can engage at end-of-decode.
//   * REPLACED `generate_stream_gemma4_once_slot_aware`'s
//     iter-2-decode-C sampling-clamp with the FULL streaming sampler/
//     grammar/stop-strings/logprobs/reasoning-text surface (Delta
//     events kind-routed by ReasoningSplitter; Logprobs events
//     emitted per-token; stop_strings terminate before final Done).
//     Sub-deferred: streaming tool-call body emission (typed
//     CapabilityUnsupported naming
//     `iter-B4c-kernel-iter-2-decode-C-stream-tool-call`).
//   * REPLACED `generate_gemma4_once_with_soft_tokens_slot_aware`'s
//     iter-2-decode-C sampling-clamp with the FULL non-streaming
//     sampler/grammar/stop-strings/logprobs/reasoning-text surface
//     (mirror of Generate-arm; the soft-token difference is fully
//     consumed upstream by the prefill call).
//   * NEW `adr040_phase_b_iter_b4c_kernel_iter2_decode_c_gemma4_tests`
//     module with H130-H136 (skip-mode source-grep pins).
//   * REVISED H87 / H125 / H126 / H127 are NOT touched — H125/H126/H127
//     pin removal of the iter-2-decode-A literal label (already removed
//     in iter-2-decode-A so the test still passes by H85 transitivity);
//     H87 still pins surviving sub-deferral labels (iter-2-decode-C is
//     now used as `iter-B4c-kernel-iter-2-decode-C-stream-tool-call`
//     for the streaming sub-deferral, so the substring
//     `iter-B4c-kernel-iter-2-decode-C per ADR-040 §6.1.38` is preserved
//     as a substring within the new label literal NO — it is replaced;
//     H136 pins the new surviving sub-deferral label).
//
// Sub-deferrals (typed CapabilityUnsupported labels):
//   * iter-B4c-kernel-iter-2-decode-C-stream-tool-call: streaming
//     tool-call body emission via ToolCallStreamEmitter at SlotId(N>0).
//     Surfaced from the GenerateStream-arm sampler entry when the
//     request engages a ToolCallSplitter.  Mirrors Wave 3 W-B3's
//     ~200 LOC incremental-arguments JSON parser; deferred so the
//     scope of iter-2-decode-C remains structurally bounded.
//
// Tests (H130-H136):
//   H130 (skip-mode): Generate orchestrator sampling-clamp REMOVED;
//                     `sampler_pure::sample_token` (or sampler chain
//                     marker) present in body.
//   H131 (skip-mode): Generate orchestrator grammar runtime construction
//                     + `mask_invalid_tokens` + `accept_bytes` calls
//                     present in body.  Grammar IS applicable to
//                     Gemma 4 (NOT N/A).
//   H132 (skip-mode): GenerateStream orchestrator `hit_stop_string` +
//                     stop_strings handling present in body.
//   H133 (skip-mode): Generate orchestrator `sample_token_with_logprob`
//                     present in body; GenerationResult.logprobs is
//                     populated (not always None).
//   H134 (skip-mode): Generate orchestrator reasoning text routing via
//                     `split_full_output` present in body; uses
//                     `registration` (NOT `_registration` underscore).
//   H135 (skip-mode): SerialFifo byte-equivalence preserved — sibling
//                     `forward_decode` signature in gemma4/forward_gpu.rs
//                     STILL contains NO slot_id / multi_seq_kv params
//                     (mirror of H128).  iter-2-decode-C is purely
//                     additive to the slot-aware orchestrator bodies;
//                     sibling fn signatures are untouched.
//   H136 (skip-mode): Qwen35 + Qwen3VL + Embed-arm UNCHANGED; surviving
//                     sub-deferral label
//                     `iter-B4c-kernel-iter-2-decode-C-stream-tool-call
//                     per ADR-040 §6.1.39` IS present (operator-grep'able
//                     pin for the streaming tool-call defer); ADR-040
//                     §6.1.39 closure block exists.

#[cfg(test)]
mod adr040_phase_b_iter_b4c_kernel_iter2_decode_c_gemma4_tests {
    // Skip-mode source-grep tests; intentionally NO `use super::*;`.

    /// **H130 (skip-mode)** — Generate orchestrator sampling-clamp
    /// REPLACED with real sampler chain.  The iter-2-decode-A
    /// `params.temperature > 0.0` sampling-clamp typed-error path is
    /// REMOVED, and the orchestrator body calls
    /// `sampler_pure::sample_token` (or the with-logprob variant) at
    /// least once.
    #[test]
    fn h130_generate_orchestrator_sampler_chain_wired() {
        let src = include_str!("engine.rs");
        let fn_marker = "fn generate_gemma4_once_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H130: generate_gemma4_once_slot_aware not found");
        // ADR-040 iter-2C + iter-2D (§6.1.46) — window bumped from
        // 30_000 to 80_000 to cover both the hybrid + HB-encoded
        // branches now that the dense F32 + legacy 4-bit branches
        // sit before them.
        let fn_window = &src[fn_idx..(fn_idx + 80_000).min(src.len())];
        // (a) OLD iter-2-decode-A sampling-clamp typed-error literal REMOVED.
        let old_label =
            "gemma4-forward-decode-slot-N-sampler-grammar (iter-B4c-kernel-iter-2-decode-C per ADR-040 §6.1.38";
        assert!(
            !fn_window.contains(old_label),
            "H130 FALSIFIED: Generate orchestrator body still contains \
             the iter-2-decode-A sampling-clamp typed-error label \
             `{old_label}`. iter-2-decode-C did not actually wire the \
             sampler chain — non-greedy requests still surface \
             CapabilityUnsupported."
        );
        // (b) sampler_pure entrypoint called from the Generate-arm body.
        assert!(
            fn_window.contains("sampler_pure::sample_token"),
            "H130 FALSIFIED: Generate orchestrator body does NOT call \
             `sampler_pure::sample_token`. iter-2-decode-C sampler \
             chain missing — non-greedy decode would fall through to \
             the on-GPU greedy argmax silently."
        );
    }

    /// **H131 (skip-mode)** — Generate orchestrator grammar wiring
    /// landed.  Gemma 4 supports grammar (Wave 2.5 W-α5 lazy grammar
    /// via ToolCallSplitter on per-model markers); iter-2-decode-C
    /// MUST wire the grammar runtime + per-token mask + accept_bytes.
    ///
    /// (a) `GrammarRuntime::new(` runtime construction present.
    /// (b) `mask::mask_invalid_tokens(` mask call present.
    /// (c) `accept_bytes(` advance call present.
    #[test]
    fn h131_generate_orchestrator_grammar_wired() {
        let src = include_str!("engine.rs");
        let fn_marker = "fn generate_gemma4_once_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H131: generate_gemma4_once_slot_aware not found");
        // ADR-040 iter-2C + iter-2D (§6.1.46) — window bumped from
        // 30_000 to 80_000 to cover both the hybrid + HB-encoded
        // branches now that the dense F32 + legacy 4-bit branches
        // sit before them.
        let fn_window = &src[fn_idx..(fn_idx + 80_000).min(src.len())];
        for required in [
            "GrammarRuntime::new(",
            "mask::mask_invalid_tokens(",
            ".accept_bytes(",
        ] {
            assert!(
                fn_window.contains(required),
                "H131 FALSIFIED: Generate orchestrator body missing \
                 grammar wiring `{required}`. iter-2-decode-C did not \
                 wire the grammar surface — grammar-constrained \
                 decode at SlotId(N>0) is non-functional."
            );
        }
    }

    /// **H132 (skip-mode)** — GenerateStream orchestrator stop_strings
    /// handling landed.  The streaming arm calls `hit_stop_string`
    /// against `params.stop_strings` and breaks the decode loop on
    /// match.
    #[test]
    fn h132_generate_stream_orchestrator_stop_strings_wired() {
        let src = include_str!("engine.rs");
        let fn_marker = "fn generate_stream_gemma4_once_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H132: generate_stream_gemma4_once_slot_aware not found");
        let fn_window = &src[fn_idx..(fn_idx + 40_000).min(src.len())];
        // (a) OLD iter-2-decode-A streaming sampling-clamp typed-error literal REMOVED.
        let old_label =
            "gemma4-forward-decode-stream-slot-N-sampler-grammar (iter-B4c-kernel-iter-2-decode-C per ADR-040 §6.1.38";
        assert!(
            !fn_window.contains(old_label),
            "H132 FALSIFIED: GenerateStream orchestrator body still \
             contains the iter-2-decode-A streaming sampling-clamp \
             typed-error label `{old_label}`. iter-2-decode-C did not \
             wire the streaming sampler/stop-strings/grammar surface."
        );
        // (b) hit_stop_string + params.stop_strings present in body.
        assert!(
            fn_window.contains("hit_stop_string("),
            "H132 FALSIFIED: GenerateStream orchestrator body does NOT \
             call `hit_stop_string`. Stop-string termination broken \
             at SlotId(N>0) — clients setting stop_strings would \
             never see early-stop semantics."
        );
        assert!(
            fn_window.contains("params.stop_strings"),
            "H132 FALSIFIED: GenerateStream orchestrator body does NOT \
             reference `params.stop_strings`. Stop-string surface \
             missing from the streaming arm at SlotId(N>0)."
        );
    }

    /// **H133 (skip-mode)** — Generate orchestrator logprobs wiring
    /// landed.  Calls `sampler_pure::sample_token_with_logprob` and
    /// the GenerationResult `logprobs:` field is populated from a
    /// non-trivial accumulator (NOT hardcoded `logprobs: None`).
    #[test]
    fn h133_generate_orchestrator_logprobs_wired() {
        let src = include_str!("engine.rs");
        let fn_marker = "fn generate_gemma4_once_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H133: generate_gemma4_once_slot_aware not found");
        // ADR-040 iter-2C + iter-2D (§6.1.46) — window bumped from
        // 30_000 to 80_000 to cover both the hybrid + HB-encoded
        // branches now that the dense F32 + legacy 4-bit branches
        // sit before them.
        let fn_window = &src[fn_idx..(fn_idx + 80_000).min(src.len())];
        assert!(
            fn_window.contains("sample_token_with_logprob"),
            "H133 FALSIFIED: Generate orchestrator body does NOT call \
             `sample_token_with_logprob`. Logprobs requests at \
             SlotId(N>0) would not get per-token logprobs."
        );
        // The previous iter-2-decode-A pinned `logprobs: None,` literal —
        // iter-2-decode-C replaces it with a non-trivial expression
        // sourced from the logprobs accumulator.  We pin the negative
        // assertion: the literal `logprobs: None,` is REMOVED from the
        // Generate orchestrator body.
        assert!(
            !fn_window.contains("logprobs: None,"),
            "H133 FALSIFIED: Generate orchestrator body still hard-codes \
             `logprobs: None,` in its GenerationResult build.  \
             iter-2-decode-C did not actually wire the logprobs \
             accumulator into the result surface."
        );
    }

    /// **H134 (skip-mode)** — Generate orchestrator reasoning-text
    /// wiring landed.  Calls `split_full_output(reg, &decoded_text)`
    /// at end-of-decode and routes the (content, reasoning) tuple
    /// into the GenerationResult.
    ///
    /// (a) The `_registration` underscore-prefix is LIFTED to
    ///     `registration` (the param is actually used).
    /// (b) `split_full_output` call present in the body.
    /// (c) `reasoning_text:` field populated from the split (not
    ///     hardcoded None).
    #[test]
    fn h134_generate_orchestrator_reasoning_text_wired() {
        let src = include_str!("engine.rs");
        let fn_marker = "fn generate_gemma4_once_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H134: generate_gemma4_once_slot_aware not found");
        // Look at fn signature window first.
        let sig_window = &src[fn_idx..(fn_idx + 1500).min(src.len())];
        assert!(
            !sig_window.contains("_registration: Option<&super::registry::ModelRegistration>"),
            "H134 FALSIFIED: Generate orchestrator signature still has \
             `_registration` (underscore prefix means unused). \
             iter-2-decode-C must lift it to `registration` to wire \
             the reasoning splitter + tool-call splitter."
        );
        // And the body window.
        // ADR-040 iter-2C + iter-2D (§6.1.46) — window bumped from
        // 30_000 to 80_000 to cover both the hybrid + HB-encoded
        // branches now that the dense F32 + legacy 4-bit branches
        // sit before them.
        let fn_window = &src[fn_idx..(fn_idx + 80_000).min(src.len())];
        // iter-230 B renamed the call to `split_full_output_forced(` (the
        // forced-open-seeded variant); accept either spelling — the pin's
        // intent is that reasoning-text routing is wired at all.
        assert!(
            fn_window.contains("split_full_output(")
                || fn_window.contains("split_full_output_forced("),
            "H134 FALSIFIED: Generate orchestrator body does NOT call \
             `split_full_output`/`split_full_output_forced`. Reasoning-\
             text routing is missing — reasoning-mode requests at \
             SlotId(N>0) would not get the reasoning_content slot \
             populated."
        );
        // The previous iter-2-decode-A pinned `reasoning_text: None,` —
        // iter-2-decode-C replaces with a non-trivial expression.
        assert!(
            !fn_window.contains("reasoning_text: None,"),
            "H134 FALSIFIED: Generate orchestrator body still hard-codes \
             `reasoning_text: None,` in its GenerationResult build. \
             iter-2-decode-C did not wire the reasoning splitter."
        );
    }

    /// **H135 (skip-mode)** — SerialFifo byte-equivalence preserved at
    /// the sibling-fn signature level (mirror of H128 carried forward
    /// to iter-2-decode-C).  The sibling `forward_decode` in
    /// `gemma4/forward_gpu.rs` MUST NOT contain `slot_id` or
    /// `multi_seq_kv*` in its signature.
    ///
    /// iter-2-decode-C touches the orchestrator bodies only — the
    /// model fn `forward_decode_slot_aware` from iter-2-decode-A is
    /// UNCHANGED (additive).  The sibling `forward_decode` REMAINS
    /// the byte-equivalence pin for SerialFifo + SlotId(0).
    #[test]
    fn h135_serial_fifo_sibling_forward_decode_signature_unchanged() {
        let src = include_str!("../../inference/models/gemma4/forward_gpu.rs");
        let sibling_marker = "pub fn forward_decode(";
        let sib_idx = src
            .find(sibling_marker)
            .expect("H135: sibling forward_decode signature missing");
        let sig_end = src[sib_idx..]
            .find(") -> Result<u32>")
            .map(|off| sib_idx + off + ") -> Result<u32>".len())
            .unwrap_or(sib_idx + 600);
        let sig_window = &src[sib_idx..sig_end.min(src.len())];
        assert!(
            !sig_window.contains("slot_id"),
            "H135 FALSIFIED: sibling `forward_decode` signature contains \
             `slot_id`. iter-2-decode-C discipline broken — sibling \
             fn signature MUST remain unchanged from iter-2-decode-A."
        );
        assert!(
            !sig_window.contains("multi_seq_kv"),
            "H135 FALSIFIED: sibling `forward_decode` signature mentions \
             `multi_seq_kv`. iter-2-decode-C discipline broken — \
             SerialFifo decode path MUST NOT consume the multi-seq \
             scaffold."
        );
        // Also: iter-2-decode-A's `forward_decode_slot_aware` signature
        // MUST still be present (iter-2-decode-C is additive to the
        // orchestrators, NOT to the model fn).
        let pf_src = include_str!("../forward_prefill.rs");
        assert!(
            pf_src.contains("pub fn forward_decode_slot_aware("),
            "H135 FALSIFIED: iter-2-decode-A's `forward_decode_slot_aware` \
             signature is missing from forward_prefill.rs. \
             iter-2-decode-C accidentally removed the load-bearing \
             primitive — orchestrator bodies have nothing to call."
        );
    }

    /// **H136 (skip-mode)** — Orthogonal surfaces UNCHANGED.  Qwen35 +
    /// Qwen3VL + Embed-arm (iter-4) lift fns + their worker_run call
    /// sites are PRESERVED.  Surviving sub-deferral label
    /// `iter-B4c-kernel-iter-2-decode-C-stream-tool-call per ADR-040
    /// §6.1.39` is present in engine.rs as an operator-grep'able pin
    /// for the streaming tool-call defer.  ADR-040 §6.1.39 closure
    /// block exists in the ADR.
    #[test]
    fn h136_orthogonal_surfaces_unchanged_and_sub_deferrals_named() {
        let src = include_str!("engine.rs");
        // Qwen35 lift fns still defined.
        for qwen35_fn in [
            "generate_qwen35_once_slot_aware(",
            "generate_stream_qwen35_once_extended_slot_aware(",
            "embed_qwen35_slot_aware(",
            "generate_qwen35_once_with_soft_tokens_slot_aware(",
        ] {
            assert!(
                src.contains(qwen35_fn),
                "H136 FALSIFIED: Qwen35 slot-aware fn `{qwen35_fn}` is \
                 NOT present in engine.rs. iter-2-decode-C accidentally \
                 regressed a Qwen35 lift — TERMINAL Qwen35 arc must be \
                 preserved."
            );
        }
        // Gemma 4 iter-1/3/4/5 lift fns still defined.
        for gemma_fn in [
            "fn generate_gemma4_once_slot_aware(",
            "fn generate_stream_gemma4_once_slot_aware(",
            "fn embed_gemma4_slot_aware(",
            "fn generate_gemma4_once_with_soft_tokens_slot_aware(",
        ] {
            assert!(
                src.contains(gemma_fn),
                "H136 FALSIFIED: Gemma 4 iter-1/3/4/5 lift fn `{gemma_fn}` \
                 is NOT defined. iter-2-decode-C accidentally regressed \
                 a prior iter's lift surface."
            );
        }
        // Embed-arm has NO decode loop (iter-4 §6.1.36 closure).
        let embed_marker = "fn embed_gemma4_slot_aware(";
        let embed_idx = src
            .find(embed_marker)
            .expect("H136: embed_gemma4_slot_aware not found");
        let embed_window = &src[embed_idx..(embed_idx + 10_000).min(src.len())];
        assert!(
            !embed_window.contains(".forward_decode_slot_aware("),
            "H136 FALSIFIED: Embed-arm fn body calls \
             `forward_decode_slot_aware`. The Embed-arm has NO decode \
             loop — calling forward_decode_slot_aware would corrupt \
             the L2-normalized embedding vector at norm_out."
        );
        // Surviving sub-deferral label for the streaming tool-call defer.
        let stream_tc_label =
            "iter-B4c-kernel-iter-2-decode-C-stream-tool-call per ADR-040 §6.1.39";
        assert!(
            src.contains(stream_tc_label),
            "H136 FALSIFIED: surviving sub-deferral label \
             `{stream_tc_label}` is NOT present in engine.rs. \
             iter-2-decode-C's streaming tool-call defer must be \
             operator-grep'able + future-iter-grep'able."
        );
        // ADR-040 §6.1.39 closure block exists.
        let adr = crate::serve::api::engine::adr040_history_doc() /* iter-230 A1: §6.1.x moved to history (aeb6e87c) */;
        assert!(
            adr.contains("### 6.1.39"),
            "H136 FALSIFIED: ADR-040 §6.1.39 closure block not found. \
             iter-2-decode-C's sub-deferral cite points at a \
             non-existent destination."
        );
    }
}

// ───────────────────────────────────────────────────────────────────
// ADR-040 Phase B iter-B4c-kernel iter-2A-cont + iter-2-decode-B —
// Gemma 4 HB-encoded (HF2Q_HYBRID_KV=0 opt-out) prefill + decode
// slot routing JOINTLY landed.
// ───────────────────────────────────────────────────────────────────
//
// Background — what was deferred pre-this-iter:
//   * iter-2A (§6.1.32, commit hash recorded at commit time) shipped
//     the 4-way dispatch fork inside the new fn
//     `MlxModelWeights::forward_prefill_with_soft_tokens_slot_aware`.
//     The HB-encoded branch (HF2Q_HYBRID_KV=0 AND cb_bits >= 5 AND
//     HF2Q_USE_DENSE=0) surfaced typed
//     `MultiSeqError::CapabilityUnsupported { capability: "...iter-
//     B4c-kernel-iter-2A-cont per ADR-040 §6.1.32..." }`.
//   * iter-2-decode-A (§6.1.38) shipped the production-default decode
//     slot routing via `forward_decode_slot_aware`'s hybrid branch.
//     The HB-encoded decode branch surfaced typed
//     `MultiSeqError::CapabilityUnsupported { capability: "...iter-
//     B4c-kernel-iter-2-decode-B per ADR-040 §6.1.38..." }`.
//
// iter-2A-cont + iter-2-decode-B (THIS iter, jointly per the brief's
// joint-iter framing) REPLACE both typed-error branches with the same
// slice_view mount + delegate-to-sibling pattern iter-2B + iter-2-decode-A
// established for the HF2Q_HYBRID_KV=1 production-default regime — now
// applied to the HF2Q_HYBRID_KV=0 opt-out HB-encoded regime, on
// `MultiSeqHbKvBuffers` instead of `MultiSeqHybridKvBuffers`.
//
// Production-code changes:
//   * `src/serve/forward_prefill.rs`:
//     - `forward_prefill_with_soft_tokens_slot_aware`: HB-encoded
//       branch (the final code path after all 3 prior branches
//       short-circuit) REPLACED typed CapabilityUnsupported with real
//       per-layer slot-view construction for the 4 buffers (K_packed
//       U8, K_norms F32, V_packed U8, V_norms F32) + mount on
//       `self.leg_hb_encoded` + delegate to
//       `forward_prefill_with_soft_tokens_resume` + restore on exit.
//     - Prefill alloc gate at line ~880 ALIGNED with decode-path gate
//       at `gemma4/forward_gpu.rs:427` via additive
//       `self.leg_hb_encoded.is_none()` predicate (mirror of iter-2B's
//       hybrid-branch alignment at line ~842).  SerialFifo byte-
//       equivalence preserved: SerialFifo enters with
//       `self.leg_hb_encoded == None`, gate fires identically.
//     - `forward_decode_slot_aware`: HB-encoded branch REPLACED typed
//       CapabilityUnsupported with real per-layer slot-view
//       construction + mount on `self.leg_hb_encoded` + delegate to
//       `forward_decode` + restore on exit.  Decode-side sibling's
//       alloc gate at `gemma4/forward_gpu.rs:427` ALREADY has
//       `&& self.leg_hb_encoded.is_none()` discipline (pre-dates this
//       iter; iter-2A-cont prefill mirrors it).
//
// No orchestrator (engine.rs) changes are needed: the orchestrators
// (`generate_gemma4_once_slot_aware` + `generate_stream_gemma4_once_
// slot_aware` + `generate_gemma4_once_with_soft_tokens_slot_aware`)
// already pass `multi_seq_kv: &mut Vec<MultiSeqHbKvBuffers>` to the
// model fns since iter-2A — that param is what the new HB-encoded
// branch routing slices into.  The `multi_seq_kv_hybrid` Option<>
// sibling param remains independently consumed by the iter-2B hybrid
// branch (it is None when HF2Q_HYBRID_KV=0, present when =1).
//
// Tests (H174-H180):
//   H174 (skip-mode): forward_prefill_with_soft_tokens_slot_aware
//                     HB-encoded branch typed-error label REMOVED;
//                     positive pin on slice_view + leg_hb_encoded
//                     mount in the new fn body.
//   H175 (skip-mode): forward_decode_slot_aware HB-encoded branch
//                     typed-error label REMOVED; positive pin on
//                     slice_view + leg_hb_encoded mount + delegate to
//                     forward_decode + restore.
//   H176 (skip-mode): HbKvBuffers slot-view construction wraps all 4
//                     buffers (k_packed / k_norms / v_packed / v_norms)
//                     in both prefill + decode bodies — per-slot
//                     isolation surface.
//   H177 (skip-mode): slot-view byte-offset arithmetic matches
//                     HbKvBuffers layout: packed (U8, 1 byte/elem)
//                     uses no `* 2` multiplier; norms (F32, 4 bytes/
//                     elem) DOES use `* 4u64`.  Defends against
//                     accidentally reusing the iter-2B F16-K
//                     `* 2u64` multiplier on the U8 K_packed buffer.
//   H178 (skip-mode): SerialFifo + HF2Q_HYBRID_KV=0 byte-equivalence
//                     preserved.  (a) Sibling fn
//                     `forward_prefill_with_soft_tokens_resume`
//                     signature UNCHANGED (no slot_id / multi_seq_kv
//                     params — mirror of H86).  (b) Prefill alloc gate
//                     at line ~880 contains `self.leg_hb_encoded.is_none()`
//                     (aligned with decode-path gate).
//   H179 (skip-mode): iter-2B + iter-2-decode-A production-default
//                     HF2Q_HYBRID_KV=1 surfaces UNCHANGED — H97 /
//                     H101 / H123 / H124 source-grep substrings
//                     still hold (positive transitivity from this
//                     iter's purely additive HB-encoded routing).
//   H180 (skip-mode): Qwen35 + Qwen3VL + Gemma 4 Embed-arm UNCHANGED;
//                     iter-1/2A/2B/3/4/5/2-decode-A/2-decode-C lift
//                     scaffolds + iter-A2b-cont / B4d Qwen35 surfaces
//                     PRESERVED.  Defense-in-depth against accidental
//                     regression at orthogonal worker arms.

#[cfg(test)]
mod adr040_phase_b_iter_b4c_kernel_iter2a_cont_iter2_decode_b_gemma4_tests {
    // Skip-mode source-grep tests; intentionally NO `use super::*;`.

    /// **H174 (skip-mode)** — `forward_prefill_with_soft_tokens_slot_aware`
    /// HB-encoded branch typed-error label REPLACED with real slot routing.
    ///
    /// iter-2A surfaced `MultiSeqError::CapabilityUnsupported { capability:
    /// "gemma4-forward-prefill-slot-N-hb-encoded (iter-B4c-kernel-iter-2A-cont
    /// per ADR-040 §6.1.32 ..." }` at every entry into the HB-encoded branch
    /// (HF2Q_HYBRID_KV=0 + cb_bits>=5 + HF2Q_USE_DENSE=0); iter-2A-cont
    /// REMOVES that typed-error capability string from the branch's
    /// `MultiSeqError::CapabilityUnsupported {` constructor + replaces with
    /// real `.slice_view(` + `self.leg_hb_encoded = Some(slot_view_hb)`
    /// mount.
    ///
    /// Note: the iter-2A-cont label substring is preserved as a doc-comment
    /// cite (the `iter-B4c-kernel-iter-2A-cont per ADR-040 §6.1.32`
    /// substring remains in the new fn body for H87 forward-pointer
    /// discoverability); the load-bearing pin is that the substring is
    /// NOT present inside a `MultiSeqError::CapabilityUnsupported { capability:`
    /// constructor call — the typed error is GONE.
    #[test]
    fn h174_iter2a_cont_hb_encoded_branch_typed_error_replaced_with_slot_routing() {
        let src = include_str!("../forward_prefill.rs");
        let fn_marker = "pub fn forward_prefill_with_soft_tokens_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H174: new fn marker present (H84 asserts)");
        // ADR-040 iter-B4c-kernel iter-2C + iter-2D (§6.1.46) — window
        // bumped from 40K to 80K to accommodate the dense F32 + legacy
        // 4-bit slot routing bodies added between the iter-2A bounds-
        // first preflight and the iter-2A-cont HB-encoded body.
        let fn_window = &src[fn_idx..(fn_idx + 80_000).min(src.len())];
        // The iter-2A HB-encoded branch typed-error capability literal —
        // the EXACT string a CapabilityUnsupported { capability: "..." }
        // constructor would have used.  iter-2A-cont REMOVES it.
        let iter2a_cont_typed_error =
            "gemma4-forward-prefill-slot-N-hb-encoded (iter-B4c-kernel-iter-2A-cont per";
        assert!(
            !fn_window.contains(iter2a_cont_typed_error),
            "H174 FALSIFIED: new fn body still contains the iter-2A HB-\
             encoded typed-error capability label `{iter2a_cont_typed_error}` \
             — iter-2A-cont slot routing NOT landed; HF2Q_HYBRID_KV=0 \
             requests still surface CapabilityUnsupported at the HB-\
             encoded branch."
        );
        // Positive pin: the iter-2A-cont label substring IS preserved
        // somewhere in the fn body (operator-grep'able forward pointer
        // — required by H87).  Either as doc-comment cite OR as the
        // iter-2A-cont sub-deferral the NEW landing might still name.
        assert!(
            fn_window.contains("iter-B4c-kernel-iter-2A-cont per ADR-040 §6.1.32"),
            "H174 FALSIFIED: new fn body does NOT contain the operator-\
             grep'able label substring `iter-B4c-kernel-iter-2A-cont per \
             ADR-040 §6.1.32`. H87 forward-pointer discoverability broken \
             — even after iter-2A-cont SHIP the substring should remain \
             as a doc-comment cite."
        );
        // Positive pin: the slot-view mount via slice_view IS present.
        assert!(
            fn_window.contains(".slice_view("),
            "H174 FALSIFIED: new fn body does NOT contain `.slice_view(` \
             — slot-view mount primitive missing."
        );
        // Positive pin: `self.leg_hb_encoded = Some(` mount IS present
        // (load-bearing for the HB-encoded slot routing — the
        // delegate-to-sibling pattern requires the sibling to read
        // `self.leg_hb_encoded`).
        assert!(
            fn_window.contains("self.leg_hb_encoded = Some("),
            "H174 FALSIFIED: new fn body does NOT contain \
             `self.leg_hb_encoded = Some(` mount — per-slot routing \
             through HbKvBuffers' slot region cannot work; the sibling \
             would see `None` and lazy-allocate a fresh single-seq \
             buffer, defeating the multi-seq scaffold."
        );
    }

    /// **H175 (skip-mode)** — `forward_decode_slot_aware` HB-encoded
    /// branch typed-error label REPLACED with real decode slot routing.
    ///
    /// Mirror of H174 for the decode body: iter-2-decode-A surfaced
    /// `MultiSeqError::CapabilityUnsupported { capability:
    /// "gemma4-forward-decode-slot-N-hb-encoded (iter-B4c-kernel-iter-
    /// 2-decode-B per ADR-040 §6.1.38 ..." }`; iter-2-decode-B REMOVES
    /// the typed error + lands the real slice_view + mount + delegate
    /// + restore pattern through `forward_decode`.
    #[test]
    fn h175_iter2_decode_b_hb_encoded_branch_typed_error_replaced_with_slot_routing() {
        let src = include_str!("../forward_prefill.rs");
        let fn_marker = "pub fn forward_decode_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H175: forward_decode_slot_aware not found (H123 asserts)");
        // ADR-040 iter-B4c-kernel iter-2-decode-D (§6.1.46) — window
        // bumped from 40K to 80K to accommodate the dense F32 + legacy
        // 4-bit decode-side slot routing bodies added between the
        // iter-2-decode-A bounds-first preflight and the iter-2-decode-B
        // HB-encoded body.
        let fn_window = &src[fn_idx..(fn_idx + 80_000).min(src.len())];
        // The iter-2-decode-A HB-encoded branch typed-error capability
        // literal.  iter-2-decode-B REMOVES it.
        let iter2_decode_b_typed_error =
            "gemma4-forward-decode-slot-N-hb-encoded (iter-B4c-kernel-iter-2-decode-B per";
        assert!(
            !fn_window.contains(iter2_decode_b_typed_error),
            "H175 FALSIFIED: decode fn body still contains the iter-2-\
             decode-A HB-encoded typed-error label `{iter2_decode_b_typed_error}` \
             — iter-2-decode-B slot routing NOT landed."
        );
        // Positive pin: label substring preserved as doc-comment cite.
        assert!(
            fn_window.contains("iter-B4c-kernel-iter-2-decode-B per ADR-040 §6.1.38"),
            "H175 FALSIFIED: decode fn body does NOT contain operator-\
             grep'able substring `iter-B4c-kernel-iter-2-decode-B per \
             ADR-040 §6.1.38`. H87 forward-pointer discoverability broken."
        );
        // Positive pin: slice_view (multiple — both hybrid + HB branches
        // mount slot-views; at least 1 of the slot-view ops is in the HB
        // branch).
        let slice_view_count = fn_window.matches(".slice_view(").count();
        assert!(
            slice_view_count >= 8, // 4 buffers per branch (hybrid + HB) × 2 mounts
            "H175 FALSIFIED: decode fn body has only {slice_view_count} \
             `.slice_view(` call(s); expected at least 8 (4 HB buffers + \
             4 hybrid buffers).  HB-encoded slice_view mount missing."
        );
        // Positive pin: `self.leg_hb_encoded = Some(` mount IS present
        // in the decode body.
        assert!(
            fn_window.contains("self.leg_hb_encoded = Some("),
            "H175 FALSIFIED: decode fn body does NOT contain \
             `self.leg_hb_encoded = Some(` mount — decode-side per-slot \
             routing through HbKvBuffers' slot region cannot work."
        );
        // Positive pin: delegate to the sibling decode kernel + restore.
        // ADR-040 S1c-2: delegate renamed to the capture-parameterized
        // `forward_decode_impl` (forward_decode_slot_aware is now a thin
        // capture_hidden=false wrapper).
        assert!(
            fn_window.contains("self.forward_decode_impl("),
            "H175 FALSIFIED: decode fn body does NOT contain \
             `self.forward_decode_impl(` delegate call. iter-2-decode-B \
             slot routing cannot reach the sibling kernel-write site."
        );
        // Positive pin: restore on exit.
        let restore_count = fn_window
            .matches("self.leg_hb_encoded = prior_leg_hb")
            .count();
        assert!(
            restore_count >= 1,
            "H175 FALSIFIED: decode fn body does NOT contain \
             `self.leg_hb_encoded = prior_leg_hb` restore. The slot-view \
             mount would leak past the call."
        );
    }

    /// **H176 (skip-mode)** — HbKvBuffers slot-view construction wraps
    /// ALL 4 buffers (k_packed / k_norms / v_packed / v_norms) in both
    /// prefill + decode bodies.  Per-slot byte isolation surface — every
    /// buffer must be sliced (not just K_packed / V_packed) or the slot
    /// routing silently shares K_norms / V_norms across slots.
    #[test]
    fn h176_hb_kv_buffers_slot_view_construction_wraps_all_four_buffers() {
        let src = include_str!("../forward_prefill.rs");
        // The new HB slot-view constructor builds `HbKvBuffers { ... }`
        // with all 4 buffers set to slot-view derivatives.  Pin the
        // 4 specific field assignments — they must appear in both the
        // prefill HB branch + decode HB branch (2 occurrences each).
        for field_marker in [
            "k_packed: k_packed_view",
            "k_norms: k_norms_view",
            "v_packed: v_packed_view",
            "v_norms: v_norms_view",
        ] {
            let count = src.matches(field_marker).count();
            assert!(
                count >= 2,
                "H176 FALSIFIED: `{field_marker}` field assignment \
                 occurs only {count} time(s) in forward_prefill.rs; \
                 expected at least 2 (one in iter-2A-cont prefill HB \
                 branch + one in iter-2-decode-B decode HB branch).  \
                 Either the prefill or decode body is missing the \
                 slot-view assignment — silently shares the buffer \
                 across slots."
            );
        }
    }

    /// **H177 (skip-mode)** — slot-view byte-offset arithmetic matches
    /// the HbKvBuffers layout.  K_packed + V_packed are U8 (1 byte/elem,
    /// NO `* 2u64` multiplier) → the byte offset arithmetic uses
    /// `packed_elems_per_slot as u64` directly with no dtype multiplier.
    /// K_norms + V_norms are F32 (4 bytes/elem) → `* 4u64` multiplier IS
    /// present.
    ///
    /// Defends against accidentally reusing the iter-2B `* 2u64` F16-K
    /// multiplier on the U8 K_packed buffer (would silently route to
    /// 2× the intended slot offset and corrupt slot 2N's region).
    #[test]
    fn h177_slice_view_byte_offset_matches_hb_kv_buffers_layout() {
        let src = include_str!("../forward_prefill.rs");
        // Anchor on the `forward_prefill_with_soft_tokens_slot_aware` fn
        // marker — covers the full prefill body including the
        // iter-2A-cont HB-encoded branch.
        let fn_marker = "pub fn forward_prefill_with_soft_tokens_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H177: forward_prefill_with_soft_tokens_slot_aware not found");
        let body_window = &src[fn_idx..(fn_idx + 50_000).min(src.len())];
        // U8 (packed) byte-offset arithmetic must use elem count directly
        // (no `* 2u64` or `* 4u64`).  Pin: the `packed_byte_offset`
        // variable name is present, and the arithmetic uses
        // `packed_elems_per_slot as u64` as the multiplier — NOT a
        // dtype-size multiplier.
        assert!(
            body_window.contains("packed_byte_offset"),
            "H177 FALSIFIED: iter-2A-cont prefill body does not bind \
             `packed_byte_offset` — slot-view K_packed/V_packed offset \
             arithmetic is missing."
        );
        // F32 (norms) byte-size MUST use `* 4u64` multiplier — the
        // canonical F32 byte-size factor mirrors iter-2B's V_norms
        // pattern at line 2850.
        assert!(
            body_window.contains("checked_mul(4u64)"),
            "H177 FALSIFIED: iter-2A-cont prefill body does not contain \
             `.checked_mul(4u64)` — F32 K_norms/V_norms byte-size \
             multiplier missing; silent-corruption regression risk \
             where norms-buffer slot offsets land at 1/4 of the right \
             byte address."
        );
    }

    /// **H178 (skip-mode)** — SerialFifo + HF2Q_HYBRID_KV=0 byte-
    /// equivalence preserved.
    ///
    /// (a) Sibling fn `forward_prefill_with_soft_tokens_resume` signature
    ///     UNCHANGED (no `slot_id` / `multi_seq_kv*` params — mirror of
    ///     H86).  Code-path disjointness: SerialFifo never reaches the
    ///     new fn (worker-arm `slot_id != SlotId(0)` predicate); SlotAware
    ///     + SlotId(0) also short-circuits.
    /// (b) Prefill alloc gate at line ~880 contains
    ///     `self.leg_hb_encoded.is_none()` — aligned with decode-path
    ///     gate at `gemma4/forward_gpu.rs:427`.  SerialFifo enters with
    ///     `None` so the gate fires identically + the legacy alloc body
    ///     runs verbatim.
    #[test]
    fn h178_serial_fifo_hf2q_hybrid_kv_zero_byte_equivalence_preserved() {
        let src = include_str!("../forward_prefill.rs");
        // (a) Sibling fn signature — no slot_id / multi_seq_kv params.
        let sib_marker = "fn forward_prefill_with_soft_tokens_resume(";
        let sib_idx = src
            .find(sib_marker)
            .expect("H178: sibling fn signature not found");
        let sig_end = src[sib_idx..]
            .find(") -> Result<u32>")
            .map(|off| sib_idx + off + ") -> Result<u32>".len())
            .unwrap_or(sib_idx + 800);
        let sig_window = &src[sib_idx..sig_end.min(src.len())];
        assert!(
            !sig_window.contains("slot_id"),
            "H178 FALSIFIED: sibling `forward_prefill_with_soft_tokens_resume` \
             signature contains `slot_id` parameter. iter-2A-cont discipline \
             broken — SerialFifo decode path must remain byte-equivalent."
        );
        assert!(
            !sig_window.contains("multi_seq_kv"),
            "H178 FALSIFIED: sibling fn signature mentions `multi_seq_kv`. \
             iter-2A-cont discipline broken — SerialFifo path must not \
             consume the multi-seq scaffold."
        );
        // (b) Prefill alloc gate aligned with decode-path discipline.
        // The legacy alloc-block scope (HF2Q_HYBRID_KV=0 path at line
        // ~880) must contain a `self.leg_hb_encoded.is_none()` predicate
        // — additive guard around the rebuild loop.
        assert!(
            src.contains("self.leg_hb_encoded.is_none()"),
            "H178 FALSIFIED: forward_prefill.rs does NOT contain \
             `self.leg_hb_encoded.is_none()` — the prefill alloc gate at \
             ~line 880 was not aligned with the decode-path gate at \
             gemma4/forward_gpu.rs:427.  SerialFifo + iter-2A-cont \
             slot-view mount would be obliterated by the unconditional \
             rebuild on the first entry into the HB branch."
        );
    }

    /// **H179 (skip-mode)** — iter-2B + iter-2-decode-A
    /// production-default HF2Q_HYBRID_KV=1 surfaces UNCHANGED.
    ///
    /// iter-2A-cont + iter-2-decode-B are purely additive to the
    /// HF2Q_HYBRID_KV=0 opt-out branches; the HF2Q_HYBRID_KV=1
    /// production-default landings stay verbatim.  Source-grep substring
    /// transitivity (H97 / H101 / H123 / H124 substrings still present).
    #[test]
    fn h179_iter2b_iter2_decode_a_production_default_surfaces_unchanged() {
        let src = include_str!("../forward_prefill.rs");
        // H97 hybrid-branch typed-error label STILL REMOVED.
        let iter2a_hybrid_typed_error =
            "gemma4-forward-prefill-slot-N-hybrid (iter-B4c-kernel-iter-2B per";
        assert!(
            !src.contains(iter2a_hybrid_typed_error),
            "H179 FALSIFIED: iter-2A hybrid-branch typed-error label \
             `{iter2a_hybrid_typed_error}` REGRESSED — iter-2A-cont \
             accidentally restored the iter-2A typed-error on the \
             production-default hybrid branch.  H97 invariant broken."
        );
        // H101 slot-view + mount pattern preserved for the hybrid branch
        // (the iter-2B hybrid-side mount is verbatim — slot_view_hybrid
        // variable name pinned).
        assert!(
            src.contains("self.hybrid_kv = Some(slot_view_hybrid)"),
            "H179 FALSIFIED: iter-2B hybrid-branch mount \
             `self.hybrid_kv = Some(slot_view_hybrid)` REGRESSED — \
             iter-2A-cont accidentally removed the iter-2B production-\
             default routing.  H101 invariant broken."
        );
        // H123 forward_decode_slot_aware fn STILL present.
        assert!(
            src.contains("pub fn forward_decode_slot_aware("),
            "H179 FALSIFIED: `forward_decode_slot_aware` fn removed — \
             iter-2-decode-B accidentally regressed iter-2-decode-A's \
             landing.  H123 invariant broken."
        );
        // H124 decode-side hybrid mount preserved.
        assert!(
            src.contains("self.hybrid_kv = Some(slot_view_hybrid)"),
            "H179 FALSIFIED: iter-2-decode-A decode hybrid mount \
             regressed.  H124 invariant broken."
        );
    }

    /// **H180 (skip-mode)** — Qwen35 + Qwen3VL + Gemma 4 Embed-arm
    /// UNCHANGED; iter-{1,2A,2B,3,4,5,2-decode-A,2-decode-C} lift
    /// scaffolds + iter-A2b-cont / B4d Qwen35 surfaces PRESERVED.
    ///
    /// Defense-in-depth against accidental regression at orthogonal
    /// worker arms — mirrors H129 + H136's structural-preservation pins.
    #[test]
    fn h180_orthogonal_surfaces_unchanged() {
        let src = include_str!("engine.rs");
        // Qwen35 lift fns must still be called from worker_run.
        for qwen35_fn in [
            "generate_qwen35_once_slot_aware(",
            "generate_stream_qwen35_once_extended_slot_aware(",
            "embed_qwen35_slot_aware(",
            "generate_qwen35_once_with_soft_tokens_slot_aware(",
        ] {
            assert!(
                src.contains(qwen35_fn),
                "H180 FALSIFIED: Qwen35 slot-aware fn call `{qwen35_fn}` \
                 is NOT present in engine.rs. iter-2A-cont / iter-2-\
                 decode-B accidentally removed a Qwen35 lift."
            );
        }
        // Gemma 4 iter-1/2A/2B/3/4/5 lift fns must still be defined.
        for gemma_fn in [
            "fn generate_gemma4_once_slot_aware(",
            "fn generate_stream_gemma4_once_slot_aware(",
            "fn embed_gemma4_slot_aware(",
            "fn generate_gemma4_once_with_soft_tokens_slot_aware(",
        ] {
            assert!(
                src.contains(gemma_fn),
                "H180 FALSIFIED: Gemma 4 iter-1/3/4/5 lift fn `{gemma_fn}` \
                 is NOT defined. iter-2A-cont / iter-2-decode-B \
                 accidentally regressed a prior iter's lift surface."
            );
        }
        // Embed-arm must NOT call forward_decode_slot_aware (mirror of
        // H129).  iter-2A-cont's reach into forward_prefill_with_soft_
        // tokens_slot_aware does NOT affect Embed — the Embed-arm calls
        // forward_embed_last, not the slot-aware prefill+decode pair.
        let embed_marker = "fn embed_gemma4_slot_aware(";
        let embed_idx = src
            .find(embed_marker)
            .expect("H180: embed_gemma4_slot_aware not found");
        let embed_window = &src[embed_idx..(embed_idx + 10_000).min(src.len())];
        assert!(
            !embed_window.contains(".forward_decode_slot_aware("),
            "H180 FALSIFIED: Embed-arm fn body calls \
             `forward_decode_slot_aware`. The Embed-arm has NO decode \
             loop — calling forward_decode_slot_aware would corrupt the \
             L2-normalized embedding vector."
        );
        // ADR-040 §6.1.45 closure block exists (forward-pin destination).
        let adr =
            crate::serve::api::engine::adr040_history_doc() /* iter-230 A1: §6.1.x moved to history (aeb6e87c) */;
        assert!(
            adr.contains("### 6.1.45"),
            "H180 FALSIFIED: ADR-040 §6.1.45 closure block not found. \
             iter-2A-cont + iter-2-decode-B sub-deferral cites point at \
             a non-existent destination."
        );
    }
}

// ─────────────────────────────────────────────────────────────────────────────
// ADR-040 iter-B4c-kernel iter-2C + iter-2D + iter-2-decode-D tests
// (H181–H188, §6.1.46, 2026-05-30).
//
// Scope decision narrative (joint shipping of 3 iters):
//
//   * iter-2C (HF2Q_TQ_CODEBOOK_BITS=4 legacy 4-bit prefill slot routing) +
//     iter-2D (HF2Q_USE_DENSE=1 dense F32 prefill slot routing) +
//     iter-2-decode-D (decode-side mirror for BOTH off-default regimes)
//     all SHIPPED jointly in this iter because:
//
//     (a) They share the SAME structural pattern: extend the model-fn
//         signatures with 2 new `Option<&mut Vec<MultiSeq{Dense,Mlx}KvBuffers>>`
//         params + thread through the 3 slot-aware orchestrators + 4
//         worker arms + extend `GemmaLoadedModel` with 2 sibling Option
//         fields + extend `provision_multi_seq_kv_for_slot_aware` with
//         Phase 3 (dense) + Phase 4 (mlx).
//
//     (b) The previously-shipped iter-2A-cont + iter-2-decode-B (§6.1.45)
//         joint-iter precedent established the operator review pattern:
//         structurally-parallel templates land in one closure block to
//         minimize cognitive load on review.
//
//     (c) Both off-default regimes ship the SAME defense-in-depth
//         scaffold-absent typed CapabilityUnsupported when the
//         iter-C2c-cont-cont Phase 3 / Phase 4 provisioning was NOT
//         engaged (the env-gate is off, so the scaffold Option is None).
//
// Background — what was deferred pre-this-iter:
//
//   * iter-2A (§6.1.32) shipped the 4-way dispatch fork in
//     `forward_prefill_with_soft_tokens_slot_aware`.  The dense F32
//     (`HF2Q_USE_DENSE=1`) branch surfaced typed
//     `iter-B4c-kernel-iter-2D per ADR-040 §6.1.32`; the legacy 4-bit
//     (`cb_bits==0`) branch surfaced typed `iter-B4c-kernel-iter-2C per
//     ADR-040 §6.1.32`.
//
//   * iter-2-decode-A (§6.1.38) shipped the decode-side mirror.  The
//     dense F32 + legacy 4-bit decode branches surfaced typed
//     `iter-B4c-kernel-iter-2-decode-D per ADR-040 §6.1.38`.
//
// iter-2C + iter-2D + iter-2-decode-D (THIS iter, jointly per the brief's
// joint-iter framing) REPLACE all 4 typed-error branches with real slot
// routing.  Two structural variants:
//
//   * iter-2D + iter-2-decode-D-dense: mount on `self.dense_kvs:
//     Option<Vec<Arc<DenseKvBuffers>>>` via slice_view ARC bundle;
//     sibling fn `forward_prefill_with_soft_tokens_resume`'s
//     `restored_lcp=None` branch gained an `is_some()` consume-gate
//     (mirror of iter-2A-cont's `self.leg_hb_encoded.is_none()` gate at
//     line ~902 + iter-2B's `self.hybrid_kv.is_none()` gate at line
//     ~860).  The decode body delegates to `forward_decode` which does
//     NOT read `self.dense_kvs` AT ALL — this is a structural fact: the
//     iter-2-decode-D-dense branch is a mount+restore preserve-strong-
//     refs operation (the TQ-active read path consumes
//     `leg_hb_encoded` / `hybrid_kv` regardless of env).
//
//   * iter-2C + iter-2-decode-D-4bit: mount on `self.kv_caches:
//     Vec<MlxKvCache>` via `std::mem::replace` of the entire Vec
//     (legacy field is always-populated at model load time per
//     `gemma4/model.rs:1292`; no Option wrapper, no is_none() gate
//     needed at sibling level).  Sibling fn body unchanged.
//
// Production-code changes:
//
//   * `src/serve/forward_prefill.rs`:
//     - 2 new imports: `MlxKvCache` (from gemma4 prelude) +
//       `MultiSeqDenseKvBuffers, MultiSeqMlxKvCache` (from
//       gemma4::kv_cache).
//     - `forward_prefill_with_soft_tokens_slot_aware`: 2 new params
//       (`multi_seq_kv_dense: Option<&mut Vec<MultiSeqDenseKvBuffers>>`
//       + `multi_seq_kv_mlx: Option<&mut Vec<MultiSeqMlxKvCache>>`).
//       Iter-2D + iter-2C branches REPLACED typed CapabilityUnsupported
//       with real per-layer slot-view construction + mount + delegate
//       + restore.
//     - `forward_decode_slot_aware`: mirror of above with same 2 new
//       params.  Iter-2-decode-D dense + 4-bit branches REPLACED typed
//       CapabilityUnsupported with real slot routing.
//     - Sibling `forward_prefill_with_soft_tokens_resume`'s
//       `restored_lcp=None` branch alloc-gate ALIGNED with the iter-2D
//       slot-aware mount discipline via additive
//       `self.dense_kvs.is_some()` consume-gate (mirror of iter-2A-cont
//       + iter-2B alloc-gate alignments).  SerialFifo byte-equivalence
//       preserved: SerialFifo enters with `self.dense_kvs == None`, gate
//       fires identically (consume branch unreachable).
//
//   * `src/serve/api/engine.rs`:
//     - `GemmaLoadedModel` extended with 2 new fields:
//       `multi_seq_kv_dense: Option<Vec<MultiSeqDenseKvBuffers>>` +
//       `multi_seq_kv_mlx: Option<Vec<MultiSeqMlxKvCache>>`.  Init to
//       None in the constructor.
//     - `provision_multi_seq_kv_for_slot_aware` extended with Phase 3
//       (dense, gated on `INVESTIGATION_ENV.use_dense`) + Phase 4
//       (mlx, gated on `cb_bits == 0`).  Off-default regimes leave the
//       respective Option as None — the model-fn defense-in-depth-fails
//       if the dispatch-fork branch is reached.
//     - 3 slot-aware orchestrator fn signatures extended with 2 new
//       `Option<&mut Vec<MultiSeq{Dense,Mlx}KvBuffers>>` params;
//       threaded through to the model-fn calls verbatim.
//     - 4 worker arms (Generate / GenerateStream / Embed / SoftTokens)
//       extended with `take`/`restore` for the 2 new fields, mirroring
//       the iter-2B / iter-3 / iter-4 / iter-5 hybrid-scaffold pattern.
//     - NEW `adr040_phase_b_iter_b4c_kernel_iter2c_iter2d_iter2_decode_d_gemma4_tests`
//       test module with H181–H188.
//
// Tests (H181–H188):
//
//   H181 (skip-mode): forward_prefill_with_soft_tokens_slot_aware
//                     iter-2C 4-bit prefill branch typed-error label
//                     REMOVED; positive pin on slot-view mount via
//                     `std::mem::replace(&mut self.kv_caches, ...)`.
//   H182 (skip-mode): forward_prefill_with_soft_tokens_slot_aware
//                     iter-2D dense F32 prefill branch typed-error label
//                     REMOVED; positive pin on slot-view mount via
//                     `self.dense_kvs = Some(slot_view_dense)`.
//   H183 (skip-mode): forward_decode_slot_aware iter-2-decode-D 4-bit
//                     decode branch typed-error label REMOVED; positive
//                     pin on `std::mem::replace(&mut self.kv_caches, ...)`.
//   H184 (skip-mode): forward_decode_slot_aware iter-2-decode-D dense
//                     decode branch typed-error label REMOVED; positive
//                     pin on `self.dense_kvs = Some(slot_view_dense)`
//                     for the decode body.
//   H185 (skip-mode): per-slot byte isolation for both layouts —
//                     `MlxKvCache` 4-buffer construction + dense F32
//                     2-buffer construction both appear at least twice
//                     in forward_prefill.rs (one prefill + one decode).
//   H186 (skip-mode): slice_view byte offsets — 4-bit uses `hd / 2` +
//                     `* 4u64` (norms F32); dense uses `dtype.size_of()`.
//   H187 (skip-mode): SerialFifo byte-equivalence preserved — sibling
//                     `forward_prefill_with_soft_tokens_resume` +
//                     `forward_decode` signatures UNCHANGED (no new
//                     params); the alloc-gate alignment via
//                     `self.dense_kvs.is_some()` consume-gate predicate
//                     short-circuits on SerialFifo (None Option).
//   H188 (skip-mode): production-default HybridKvBuffers + HB-encoded
//                     paths UNCHANGED (H179 transitivity); Qwen35 +
//                     Qwen3VL UNCHANGED.

#[cfg(test)]
mod adr040_phase_b_iter_b4c_kernel_iter2c_iter2d_iter2_decode_d_gemma4_tests {

    /// **H181 (skip-mode)** — iter-2C 4-bit prefill branch typed-error
    /// REPLACED with real slot routing.
    ///
    /// The iter-2A typed-deferral capability literal
    /// `gemma4-forward-prefill-slot-N-legacy-4bit (iter-B4c-kernel-iter-2C per`
    /// is REMOVED from the new fn body — iter-2C's slot routing has
    /// REPLACED it.  Positive pin: `std::mem::replace(&mut self.kv_caches,`
    /// IS present (the Vec-swap mount primitive for the always-populated
    /// legacy `self.kv_caches: Vec<MlxKvCache>` field).
    ///
    /// Note: the iter-2C label substring is preserved as a doc-comment
    /// cite (the `iter-B4c-kernel-iter-2C per ADR-040 §6.1.32` substring
    /// remains in the new fn body for H87 forward-pointer discoverability);
    /// the load-bearing pin is that the substring is NOT present inside
    /// a `MultiSeqError::CapabilityUnsupported { capability: "..." }`
    /// constructor call — the typed error is GONE.
    #[test]
    fn h181_iter2c_legacy_4bit_branch_typed_error_replaced_with_slot_routing() {
        let src = include_str!("../forward_prefill.rs");
        let fn_marker = "pub fn forward_prefill_with_soft_tokens_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H181: new fn marker present (H84 asserts)");
        let fn_window = &src[fn_idx..(fn_idx + 80_000).min(src.len())];
        // The iter-2A 4-bit branch typed-error capability literal —
        // EXACT string a CapabilityUnsupported constructor would have
        // used.  iter-2C REMOVES it.
        let iter2c_typed_error =
            "gemma4-forward-prefill-slot-N-legacy-4bit (iter-B4c-kernel-iter-2C per";
        assert!(
            !fn_window.contains(iter2c_typed_error),
            "H181 FALSIFIED: new fn body still contains the iter-2A \
             legacy 4-bit typed-error capability label \
             `{iter2c_typed_error}` — iter-2C slot routing NOT landed; \
             HF2Q_TQ_CODEBOOK_BITS=4 requests still surface \
             CapabilityUnsupported at the legacy 4-bit branch."
        );
        // Positive pin: the iter-2C label substring IS preserved
        // somewhere in the fn body as doc-comment cite (operator-
        // grep'able forward pointer — required by H87 transitivity).
        assert!(
            fn_window.contains("iter-B4c-kernel-iter-2C per ADR-040 §6.1.32"),
            "H181 FALSIFIED: new fn body does NOT contain the operator-\
             grep'able label substring `iter-B4c-kernel-iter-2C per \
             ADR-040 §6.1.32`. H87 forward-pointer discoverability broken \
             — even after iter-2C SHIP the substring should remain as a \
             doc-comment cite."
        );
        // Positive pin: the Vec-swap mount primitive IS present for
        // the legacy `self.kv_caches: Vec<MlxKvCache>` field (no
        // Option-wrapper; mem::replace swaps the entire Vec).
        assert!(
            fn_window.contains("std::mem::replace(&mut self.kv_caches"),
            "H181 FALSIFIED: new fn body does NOT contain \
             `std::mem::replace(&mut self.kv_caches` mount — per-slot \
             routing through MlxKvCache's slot region cannot work; the \
             sibling would see the persistent per-layer cache instead \
             of the slot-view bundle."
        );
        // Positive pin: `MlxKvCache {` construction IS present (the
        // mount path builds the legacy single-seq struct from the 4
        // slot-view buffers).
        assert!(
            fn_window.contains("MlxKvCache {"),
            "H181 FALSIFIED: new fn body does NOT construct \
             `MlxKvCache {{ ... }}` from the slot-views.  The mount path \
             must produce the legacy struct so the sibling fn's \
             `self.kv_caches[layer_idx].k_packed` read at \
             `gemma4/forward_gpu.rs:1525-1526` can read it bit-identically."
        );
    }

    /// **H182 (skip-mode)** — iter-2D dense F32 prefill branch typed-error
    /// REPLACED with real slot routing.
    ///
    /// Mirror of H181 for the dense F32 path: the iter-2A typed-deferral
    /// capability literal `gemma4-forward-prefill-slot-N-dense-F32
    /// (iter-B4c-kernel-iter-2D per` is REMOVED; positive pin on the
    /// mount via `self.dense_kvs = Some(slot_view_dense)` (mirror of
    /// iter-2B's `self.hybrid_kv = Some(...)` mount).
    #[test]
    fn h182_iter2d_dense_f32_branch_typed_error_replaced_with_slot_routing() {
        let src = include_str!("../forward_prefill.rs");
        let fn_marker = "pub fn forward_prefill_with_soft_tokens_slot_aware(";
        let fn_idx = src.find(fn_marker).expect("H182: new fn marker present");
        let fn_window = &src[fn_idx..(fn_idx + 80_000).min(src.len())];
        let iter2d_typed_error =
            "gemma4-forward-prefill-slot-N-dense-F32 (iter-B4c-kernel-iter-2D per";
        assert!(
            !fn_window.contains(iter2d_typed_error),
            "H182 FALSIFIED: new fn body still contains the iter-2A \
             dense F32 typed-error capability label \
             `{iter2d_typed_error}` — iter-2D slot routing NOT landed; \
             HF2Q_USE_DENSE=1 requests still surface CapabilityUnsupported."
        );
        // Positive pin: label substring preserved as doc-comment cite.
        assert!(
            fn_window.contains("iter-B4c-kernel-iter-2D per ADR-040 §6.1.32"),
            "H182 FALSIFIED: new fn body does NOT contain operator-\
             grep'able substring `iter-B4c-kernel-iter-2D per ADR-040 \
             §6.1.32`. H87 forward-pointer discoverability broken."
        );
        // Positive pin: the dense mount IS present.
        assert!(
            fn_window.contains("self.dense_kvs = Some(slot_view_dense)"),
            "H182 FALSIFIED: new fn body does NOT contain \
             `self.dense_kvs = Some(slot_view_dense)` mount — per-slot \
             routing through DenseKvBuffers' slot region cannot work; \
             the sibling's consume-gate at line ~676 cannot consume the \
             slot-view bundle."
        );
        // Positive pin: scaffold-absent defense-in-depth label IS
        // present (operator who flipped HF2Q_USE_DENSE post-LazyLock
        // would land here).
        assert!(
            fn_window.contains("gemma4-forward-prefill-dense-scaffold-absent"),
            "H182 FALSIFIED: new fn body does NOT contain the iter-2D \
             defense-in-depth scaffold-absent label \
             `gemma4-forward-prefill-dense-scaffold-absent`.  Operator \
             who flipped HF2Q_USE_DENSE post-spawn would surface a less-\
             informative error."
        );
    }

    /// **H183 (skip-mode)** — iter-2-decode-D 4-bit decode branch typed-
    /// error REPLACED with real slot routing.
    ///
    /// Mirror of H181 for the decode body: the iter-2-decode-A typed-
    /// deferral capability literal `gemma4-forward-decode-slot-N-legacy-
    /// 4bit (iter-B4c-kernel-iter-2-decode-D per` is REMOVED.
    #[test]
    fn h183_iter2_decode_d_4bit_branch_typed_error_replaced_with_slot_routing() {
        let src = include_str!("../forward_prefill.rs");
        let fn_marker = "pub fn forward_decode_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H183: forward_decode_slot_aware not found (H123 asserts)");
        let fn_window = &src[fn_idx..(fn_idx + 80_000).min(src.len())];
        let iter2_decode_d_4bit_error =
            "gemma4-forward-decode-slot-N-legacy-4bit (iter-B4c-kernel-iter-2-decode-D per";
        assert!(
            !fn_window.contains(iter2_decode_d_4bit_error),
            "H183 FALSIFIED: decode fn body still contains the iter-2-\
             decode-A legacy 4-bit typed-error label \
             `{iter2_decode_d_4bit_error}` — iter-2-decode-D 4-bit slot \
             routing NOT landed."
        );
        // Positive pin: Vec-swap mount via mem::replace IS present.
        assert!(
            fn_window.contains("std::mem::replace(&mut self.kv_caches"),
            "H183 FALSIFIED: decode fn body does NOT contain \
             `std::mem::replace(&mut self.kv_caches` mount — decode-side \
             per-slot routing through MlxKvCache cannot work."
        );
        // Positive pin: scaffold-absent defense-in-depth label IS present.
        assert!(
            fn_window.contains("gemma4-forward-decode-mlx-scaffold-absent"),
            "H183 FALSIFIED: decode fn body does NOT contain the iter-2-\
             decode-D defense-in-depth scaffold-absent label."
        );
    }

    /// **H184 (skip-mode)** — iter-2-decode-D dense F32 decode branch
    /// typed-error REPLACED with real slot routing.
    ///
    /// Mirror of H182 for the decode body.  Note: forward_decode does
    /// NOT consume `self.dense_kvs` at runtime; the mount+restore is
    /// structurally a no-op for the dense F32 read path (the TQ-active
    /// path routes via leg_hb_encoded / hybrid_kv).  H184 pins typed-
    /// error removal + mount construction so the byte-offset arithmetic
    /// is verified (H186) and the persistent scaffold's strong refs are
    /// preserved.
    #[test]
    fn h184_iter2_decode_d_dense_branch_typed_error_replaced_with_slot_routing() {
        let src = include_str!("../forward_prefill.rs");
        let fn_marker = "pub fn forward_decode_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H184: forward_decode_slot_aware not found");
        let fn_window = &src[fn_idx..(fn_idx + 80_000).min(src.len())];
        let iter2_decode_d_dense_error =
            "gemma4-forward-decode-slot-N-dense-F32 (iter-B4c-kernel-iter-2-decode-D per";
        assert!(
            !fn_window.contains(iter2_decode_d_dense_error),
            "H184 FALSIFIED: decode fn body still contains the iter-2-\
             decode-A dense F32 typed-error label \
             `{iter2_decode_d_dense_error}` — iter-2-decode-D dense slot \
             routing NOT landed."
        );
        // Positive pin: dense mount IS present in the decode body
        // (mount on self.dense_kvs).  forward_decode does not READ
        // self.dense_kvs but the mount preserves the persistent
        // scaffold's strong refs for future iters that may add a
        // dense F32 decode read path.
        assert!(
            fn_window.contains("self.dense_kvs = Some(slot_view_dense)"),
            "H184 FALSIFIED: decode fn body does NOT contain \
             `self.dense_kvs = Some(slot_view_dense)` mount.  The byte-\
             offset arithmetic verification + persistent scaffold strong-\
             ref preservation cannot work without the mount."
        );
        // Positive pin: scaffold-absent defense-in-depth label IS present.
        assert!(
            fn_window.contains("gemma4-forward-decode-dense-scaffold-absent"),
            "H184 FALSIFIED: decode fn body does NOT contain the iter-2-\
             decode-D defense-in-depth scaffold-absent dense label."
        );
    }

    /// **H185 (skip-mode + structural)** — per-slot byte isolation for
    /// both layouts.
    ///
    /// The 4-buffer MlxKvCache construction (`MlxKvCache { k_packed:`,
    /// `k_norms:`, `v_packed:`, `v_norms:`) appears in BOTH prefill +
    /// decode bodies (≥2 each); the 2-buffer DenseKvBuffers ARC
    /// construction (`DenseKvBuffers { k:`, `v:`) appears in BOTH
    /// prefill + decode bodies (≥2 each).  Defends against silently
    /// sharing buffers across slots.
    #[test]
    fn h185_per_slot_byte_isolation_4bit_and_dense_constructs_in_prefill_and_decode() {
        let src = include_str!("../forward_prefill.rs");
        // 4-buffer MlxKvCache construction: k_packed + k_norms +
        // v_packed + v_norms — each appears in at LEAST 2 places
        // (1 prefill + 1 decode; the legacy alloc site at
        // gemma4/model.rs is in a different file).
        for required in [
            "MlxKvCache {",
            "k_packed: k_packed_view",
            "k_norms: k_norms_view",
            "v_packed: v_packed_view",
            "v_norms: v_norms_view",
        ] {
            let count = src.matches(required).count();
            assert!(
                count >= 2,
                "H185 FALSIFIED: MlxKvCache field assignment `{required}` \
                 appears {count} time(s) in forward_prefill.rs — expected \
                 ≥2 (1 prefill + 1 decode).  Per-slot isolation broken: \
                 either the prefill or decode 4-bit branch is silently \
                 sharing buffers across slots."
            );
        }
        // 2-buffer DenseKvBuffers construction: k + v.  Note the legacy
        // alloc site at line ~705 also constructs DenseKvBuffers, so
        // the expected count is ≥3 (1 legacy + 1 iter-2D prefill +
        // 1 iter-2D decode).
        for required in ["k: k_view,", "v: v_view,"] {
            let count = src.matches(required).count();
            assert!(
                count >= 2,
                "H185 FALSIFIED: DenseKvBuffers field assignment \
                 `{required}` appears {count} time(s) — expected ≥2 \
                 (1 prefill + 1 decode).  Per-slot isolation broken on \
                 the dense F32 path."
            );
        }
    }

    /// **H186 (skip-mode)** — slice_view byte offsets match the legacy
    /// layouts for both 4-bit + dense F32 variants.
    ///
    /// 4-bit: K_packed / V_packed are U8 = 1 byte/elem with shape
    /// `[nkv, cap, hd/2]`; K_norms / V_norms are F32 = 4 bytes/elem.
    /// Dense F32: K + V are `dtype.size_of()` (4 for F32, 2 for F16).
    ///
    /// Pin defends a regression where iter-2C accidentally uses
    /// iter-2B's `* 2u64` F16-K multiplier on the U8 K_packed buffer
    /// (would silently route to wrong slot).
    #[test]
    fn h186_slice_view_byte_offsets_match_4bit_and_dense_layouts() {
        let src = include_str!("../forward_prefill.rs");
        // The MLX (4-bit) slot-view uses `hd_half = hd / 2` for the
        // packed buffer's shape — that's the load-bearing structural
        // marker of the U8-packed half-nibble shape.
        assert!(
            src.contains("hd_half = hd / 2"),
            "H186 FALSIFIED: forward_prefill.rs does not contain the \
             4-bit nibble-pack shape marker `hd_half = hd / 2`.  The \
             MLX slot-view would use the wrong shape for the U8 packed \
             buffers (silently corrupting per-slot byte addressing)."
        );
        // The MLX slot-view's packed byte offset uses the U8 = 1 byte/\
        // elem discipline (`checked_mul(packed_elems_per_slot as u64)`
        // — no `* 2u64` or `* 4u64` multiplier on packed elements).
        assert!(
            src.contains("checked_mul(packed_elems_per_slot as u64) // U8 = 1 byte/elem"),
            "H186 FALSIFIED: forward_prefill.rs does not contain the \
             MLX packed byte-offset arithmetic with the U8 1-byte/elem \
             marker comment.  An accidental F16 `* 2u64` multiplier on \
             the packed buffer would silently target the wrong slot."
        );
        // The MLX norms byte offset uses `* 4u64` (F32) — same as the
        // HB norms layout (mirror of iter-2A-cont's norms_byte_offset
        // discipline).
        assert!(
            src.matches("checked_mul(4u64) // F32 = 4 bytes/elem")
                .count()
                >= 2,
            "H186 FALSIFIED: forward_prefill.rs F32 4-byte/elem norms \
             multiplier marker appears <2 times — expected ≥2 (prefill + \
             decode MLX norms slot-view arithmetic)."
        );
        // The dense slot-view uses `dtype.size_of()` (dtype-aware).
        // Mirror of iter-2B's `v_dtype_size` discipline at line ~2829.
        assert!(
            src.contains("dtype_size = dtype.size_of()"),
            "H186 FALSIFIED: forward_prefill.rs does not contain the \
             dense F32 dtype-aware byte-size lookup `dtype_size = \
             dtype.size_of()`.  An accidental hardcoded F32 multiplier \
             would corrupt slot addressing under HF2Q_F16_KV=1."
        );
    }

    /// **H187 (skip-mode)** — SerialFifo byte-equivalence preserved at
    /// slot 0 for both regimes.
    ///
    /// (a) Sibling fn `forward_prefill_with_soft_tokens_resume`'s
    ///     signature is UNCHANGED (mirror of H86).
    /// (b) Sibling fn `forward_decode`'s signature is UNCHANGED (mirror
    ///     of H128).
    /// (c) The iter-2D alloc-gate alignment added a
    ///     `self.dense_kvs.is_some()` consume-gate predicate inside the
    ///     sibling's `restored_lcp=None` branch.  SerialFifo enters
    ///     with `self.dense_kvs == None`, so the consume branch is
    ///     unreachable; the fresh-alloc body runs verbatim.
    /// (d) The iter-2C path mounts via `std::mem::replace` of
    ///     `self.kv_caches: Vec<MlxKvCache>` — the SerialFifo path
    ///     never reaches the slot-aware fn (iter-1 worker-arm predicate
    ///     `slot_id != SlotId(0)`) so the mount is unreachable on
    ///     SerialFifo.
    #[test]
    fn h187_serial_fifo_byte_equivalence_preserved_for_4bit_and_dense() {
        let src = include_str!("../forward_prefill.rs");
        // (a) Sibling fn forward_prefill_with_soft_tokens_resume's
        // signature is unchanged (no slot_id / multi_seq_kv params).
        let sibling_marker = "pub fn forward_prefill_with_soft_tokens_resume(";
        let sibling_idx = src
            .find(sibling_marker)
            .expect("H187: sibling fn forward_prefill_with_soft_tokens_resume present");
        let sibling_sig = &src[sibling_idx..(sibling_idx + 2_000).min(src.len())];
        assert!(
            !sibling_sig.contains("slot_id:"),
            "H187 FALSIFIED: sibling `forward_prefill_with_soft_tokens_resume` \
             signature contains `slot_id:` — H86 byte-equivalence pin \
             broken; SerialFifo would be routed through the slot-aware path."
        );
        assert!(
            !sibling_sig.contains("multi_seq_kv"),
            "H187 FALSIFIED: sibling fn signature contains `multi_seq_kv` \
             — H86 byte-equivalence pin broken."
        );
        // (b) Sibling fn forward_decode's signature is unchanged.
        let decode_sibling_marker = "pub fn forward_decode(";
        let decode_idx = src.find(decode_sibling_marker).or_else(|| {
            // forward_decode may live in gemma4/forward_gpu.rs;
            // check there if not in forward_prefill.rs.
            None
        });
        // Either forward_decode is in forward_prefill.rs (skip-mode pin
        // would scan its sig here) or it's in gemma4/forward_gpu.rs
        // (skip the in-file check).
        if let Some(idx) = decode_idx {
            let decode_sig = &src[idx..(idx + 2_000).min(src.len())];
            assert!(
                !decode_sig.contains("slot_id:"),
                "H187 FALSIFIED: sibling `forward_decode` signature \
                 contains `slot_id:` — H128 byte-equivalence pin broken."
            );
        }
        // (c) The iter-2D consume-gate predicate exists in the sibling
        // fn body.  SerialFifo enters with self.dense_kvs == None, so
        // this gate's consume branch is unreachable.
        assert!(
            src.contains("if self.dense_kvs.is_some()"),
            "H187 FALSIFIED: sibling fn body does NOT contain the iter-\
             2D alloc-gate alignment `if self.dense_kvs.is_some()` \
             predicate.  The slot-aware mount would be obliterated by \
             the sibling's unconditional fresh-alloc; iter-2D consume \
             discipline broken."
        );
    }

    /// **H188 (skip-mode)** — production-default + Qwen35 + Qwen3VL
    /// surfaces UNCHANGED.
    ///
    /// (a) iter-2B hybrid-branch typed-error label STILL REMOVED (H97
    ///     transitivity).
    /// (b) iter-2B `self.hybrid_kv = Some(slot_view_hybrid)` mount
    ///     STILL present (H101 transitivity).
    /// (c) iter-2-decode-A `forward_decode_slot_aware` fn STILL defined
    ///     (H123 transitivity).
    /// (d) iter-2A-cont HB-encoded branch `self.leg_hb_encoded = Some(`
    ///     mount STILL present (H174 transitivity).
    /// (e) iter-2-decode-B HB-encoded decode branch `self.leg_hb_encoded
    ///     = Some(` mount STILL present (H175 transitivity).
    /// (f) Qwen35 + Qwen3VL surfaces in engine.rs UNCHANGED.
    /// (g) ADR-040 §6.1.46 closure block exists.
    #[test]
    fn h188_production_default_and_qwen35_qwen3vl_surfaces_unchanged() {
        let pf_src = include_str!("../forward_prefill.rs");
        // (a) iter-2B hybrid-branch typed-error label STILL REMOVED.
        let iter2a_hybrid_typed_error =
            "gemma4-forward-prefill-slot-N-hybrid (iter-B4c-kernel-iter-2B per";
        assert!(
            !pf_src.contains(iter2a_hybrid_typed_error),
            "H188 FALSIFIED: iter-2A hybrid-branch typed-error label \
             `{iter2a_hybrid_typed_error}` REGRESSED — iter-2C / 2D / \
             2-decode-D accidentally restored the iter-2A typed-error \
             on the production-default hybrid branch."
        );
        // (b) iter-2B hybrid mount STILL present.
        assert!(
            pf_src.contains("self.hybrid_kv = Some(slot_view_hybrid)"),
            "H188 FALSIFIED: iter-2B hybrid-branch mount \
             `self.hybrid_kv = Some(slot_view_hybrid)` REGRESSED."
        );
        // (c) iter-2-decode-A fn STILL defined.
        assert!(
            pf_src.contains("pub fn forward_decode_slot_aware("),
            "H188 FALSIFIED: `forward_decode_slot_aware` fn removed — \
             H123 transitivity broken."
        );
        // (d) iter-2A-cont HB-encoded mount STILL present.
        assert!(
            pf_src.matches("self.leg_hb_encoded = Some(").count() >= 2,
            "H188 FALSIFIED: `self.leg_hb_encoded = Some(` mount count \
             < 2 (expected ≥2: one in prefill, one in decode).  \
             iter-2A-cont (H174) or iter-2-decode-B (H175) regression."
        );
        let src = include_str!("./engine.rs");
        // (f) Qwen35 worker-arm slot-aware fns + worker-arm structural
        // elements UNCHANGED (presence pin).
        for required in [
            "generate_qwen35_once_slot_aware",
            "generate_stream_qwen35_once_slot_aware",
        ] {
            assert!(
                src.contains(required),
                "H188 FALSIFIED: Qwen35 slot-aware surface `{required}` \
                 removed — iter-2C / 2D / 2-decode-D accidentally \
                 touched Qwen35 architecture."
            );
        }
        // (g) Embed-arm must NOT call forward_decode_slot_aware (mirror
        // of H180).
        let fn_marker = "fn embed_gemma4_slot_aware(";
        let embed_idx = src
            .find(fn_marker)
            .expect("H188: embed_gemma4_slot_aware not found");
        let embed_window = &src[embed_idx..(embed_idx + 10_000).min(src.len())];
        assert!(
            !embed_window.contains(".forward_decode_slot_aware("),
            "H188 FALSIFIED: Embed-arm fn body calls \
             `forward_decode_slot_aware`.  The Embed-arm has NO decode \
             loop — calling forward_decode_slot_aware would corrupt the \
             L2-normalized embedding vector."
        );
        // (h) ADR-040 §6.1.46 closure block exists.
        let adr =
            crate::serve::api::engine::adr040_history_doc() /* iter-230 A1: §6.1.x moved to history (aeb6e87c) */;
        assert!(
            adr.contains("### 6.1.46"),
            "H188 FALSIFIED: ADR-040 §6.1.46 closure block not found. \
             iter-2C + iter-2D + iter-2-decode-D sub-deferral cites \
             point at a non-existent destination."
        );
    }
}

// ─────────────────────────────────────────────────────────────────────
// ADR-040 Phase B iter-B4c-kernel iter-2B-xlen + iter-2-decode-A-xlen
// joint test module — Gemma 4 BF16 xlen K/V (HF2Q_DFLASH_XLEN_SDPA=1
// opt-in surface, ADR-030 iter-96) slot routing JOINTLY landed for the
// HybridKvBuffers production-default KV regime on BOTH the prefill +
// decode sides via the iter-2B + iter-2-decode-A slice_view mount +
// delegate-to-sibling pattern applied to the optional bf16_xlen_k /
// bf16_xlen_v fields.
//
// Joint-iter framing per the iter-2C + iter-2D + iter-2-decode-D
// precedent at §6.1.46: iter-2B-xlen + iter-2-decode-A-xlen share the
// SAME structural template (per-layer slot-view construction at the
// same byte-offset arithmetic — `slot_id.0 * nkv * cap * hd * 2`,
// BF16 = 2 bytes/elem) applied at TWO model-fn entry points
// (`forward_prefill_with_soft_tokens_slot_aware`'s hybrid branch +
// `forward_decode_slot_aware`'s hybrid branch).  Joint shipping
// minimizes cognitive load on operator review AND eliminates the
// surface-area drift risk that would emerge if the prefill side
// landed at iter-N while the decode side waited at iter-N+M.
//
// Path chosen — slice_view mount + delegate-to-sibling (Path A from
// §6.1.34 + §6.1.38), additive variant:
//
//   * The iter-2B + iter-2-decode-A hybrid branches already mount
//     per-layer `HybridKvBuffers` slot-views; iter-2B-xlen +
//     iter-2-decode-A-xlen REPLACE the `bf16_xlen_k: None,
//     bf16_xlen_v: None` literal in the `HybridKvBuffers {}`
//     constructor with conditional `Some(slot_view) / None` based on
//     `xlen_engaged: bool` derived from the persistent multi-seq
//     scaffold's first-layer presence check.
//   * The xlen typed-error gate at the iter-2B + iter-2-decode-A
//     branch entries is REPLACED with a presence consistency check
//     (every layer must have both bf16_xlen_k.is_some() AND
//     bf16_xlen_v.is_some() OR every layer must have BOTH as None —
//     mixed presence indicates alloc-helper corruption and bails
//     with a typed `CapabilityUnsupported` naming the inconsistency).
//   * No new fn signatures, no new GemmaLoadedModel fields, no new
//     orchestrator threading — the persistent xlen K/V buffers are
//     already inside the iter-A3b iter-1 `MultiSeqHybridKvBuffers`
//     scaffold provisioned by iter-C2c-cont at §6.1.33; this iter
//     simply consumes them on the slot-routing read path.
//
// Why this is structurally honest:
//
//   * Default OFF (`HF2Q_DFLASH_XLEN_SDPA` unset): every layer's
//     bf16_xlen_k + bf16_xlen_v are `None` (alloc-time decision per
//     `gemma4/kv_cache.rs:1102-1115`) → `xlen_engaged == false` →
//     the conditional materialization produces `(None, None)` →
//     `bf16_xlen_k: None, bf16_xlen_v: None` propagates verbatim
//     into the legacy `HybridKvBuffers` struct.  PRE-iter-2B-xlen
//     + iter-2-decode-A-xlen byte equivalence preserved (H193).
//   * Default ON: every layer's bf16_xlen_k + bf16_xlen_v are Some(_)
//     by the alloc helper's atomic alloc loop (either both alloc
//     succeeded for every layer or none did) → `xlen_engaged == true`
//     → slot-views materialize at the per-slot byte region of the
//     persistent multi-seq scaffold → sibling fn's downstream xlen
//     consumer (`dispatch_kv_cache_copy_seq_bf16_to_bf16_head_major`
//     at `forward_prefill_batched.rs:1515`) sees the slot-view ARC
//     handle just like the iter-2B F16 K + V slot-views.
//   * SerialFifo byte-equivalence (H194) preserved by code-path
//     disjointness: SerialFifo + SlotId(0) routes through
//     `generate_once` / `generate_stream_once` direct calls to the
//     unchanged siblings; the iter-1 worker-arm predicate
//     `slot_id != SlotId(0)` short-circuits the slot-aware fns at
//     entry.  iter-2B-xlen + iter-2-decode-A-xlen are observed only
//     at SlotAware + SlotId(N>0).
//
// Production-code changes (NO NEW FN SIGNATURES — purely body-additive
// to the iter-2B + iter-2-decode-A hybrid branches):
//
//   * `src/serve/forward_prefill.rs`:
//     - iter-2B hybrid branch xlen typed-error gate (~16 LOC):
//       REPLACED with `xlen_engaged: bool` derivation + presence
//       consistency invariant check (defense-in-depth typed
//       `CapabilityUnsupported` only on mixed-presence, which is
//       impossible per alloc-helper construction).
//     - iter-2B hybrid branch per-layer slot-view construction
//       (~65 LOC): NEW conditional BF16 xlen K + V slot-view block
//       computed inside the per-layer loop right before the
//       `HybridKvBuffers {}` struct literal.
//     - iter-2B hybrid branch struct literal (~2 LOC): the
//       `bf16_xlen_k: None, bf16_xlen_v: None` literal REPLACED with
//       `bf16_xlen_k: bf16_xlen_k_view, bf16_xlen_v: bf16_xlen_v_view`
//       binding to the conditional materialization.
//     - iter-2-decode-A hybrid branch: same 3 changes applied
//       verbatim to the decode body (mirror of prefill).
//
// Tests (H189–H195):
//
//   H189 (skip-mode): forward_prefill_with_soft_tokens_slot_aware
//                     iter-2B-xlen hybrid xlen branch typed-error
//                     label REMOVED; positive pin on slot-view
//                     materialization via `xlen_engaged` binding +
//                     `bf16_xlen_k: bf16_xlen_k_view` non-None
//                     propagation.
//   H190 (skip-mode): forward_decode_slot_aware iter-2-decode-A-xlen
//                     hybrid xlen branch typed-error label REMOVED;
//                     positive pin on mirror of H189 applied to
//                     decode body.
//   H191 (skip-mode): per-slot byte offset arithmetic uses
//                     `* 2u64 // BF16 = 2 bytes/elem` + references
//                     `k_elems_per_slot` (reuses the F16 K stride
//                     formula since BF16 K shape is identical).
//                     Defends against accidentally using F32 stride
//                     (`* 4u64`) or U8 stride (`* 1u64`) on the BF16
//                     buffer.
//   H192 (skip-mode): per-slot byte isolation — the `xlen_byte_offset`
//                     binding uses `slot_id.0` as the multiplier on
//                     `xlen_bytes_per_slot`.  Defends against a
//                     hardcoded 0 byte offset that would cause every
//                     slot to write to slot 0's xlen region.
//   H193 (skip-mode): default OFF (HF2Q_DFLASH_XLEN_SDPA unset) path
//                     UNCHANGED — `xlen_engaged` falls through to
//                     `(None, None)` materialization when no layer
//                     carries Some xlen buffers; the per-layer
//                     `HybridKvBuffers {}` struct receives the
//                     (None, None) literal verbatim equivalent to
//                     pre-iter-2B-xlen + iter-2-decode-A-xlen state.
//   H194 (skip-mode): SerialFifo + HF2Q_DFLASH_XLEN_SDPA=1 byte
//                     equivalence preserved — sibling fn signatures
//                     `forward_prefill_with_soft_tokens_resume` +
//                     `forward_decode` UNCHANGED (no xlen-specific
//                     params); code-path disjointness preserves H86
//                     + H128 byte equivalence.
//   H195 (skip-mode): production-default HybridKvBuffers non-xlen +
//                     HB-encoded + dense F32 + legacy 4-bit surfaces
//                     UNCHANGED (H188 transitivity); Qwen35 +
//                     Qwen3VL UNCHANGED.  ADR-040 §6.1.47 closure
//                     block exists.

#[cfg(test)]
mod adr040_phase_b_iter_b4c_kernel_iter2b_xlen_iter2_decode_a_xlen_gemma4_tests {
    // Skip-mode source-grep tests; intentionally NO `use super::*;`.

    /// **H189 (skip-mode)** — iter-2B-xlen hybrid xlen branch
    /// typed-error REPLACED with real slot routing.
    ///
    /// The iter-2B typed-deferral capability literal
    /// `gemma4-forward-prefill-slot-N-hybrid-xlen (iter-B4c-kernel-iter-2B-xlen per`
    /// is REMOVED from the new fn body — iter-2B-xlen's slot routing
    /// has REPLACED it.  Positive pin: `xlen_engaged` binding +
    /// `bf16_xlen_k: bf16_xlen_k_view,` (non-None propagation through
    /// the `HybridKvBuffers {}` constructor) BOTH present in the
    /// prefill fn body.
    ///
    /// Note: the iter-2B-xlen label substring is preserved as a
    /// doc-comment cite (the `iter-B4c-kernel-iter-2B-xlen per ADR-040 §6.1.47`
    /// substring remains in the new fn body for H87 forward-pointer
    /// discoverability); the load-bearing pin is that the
    /// substring is NOT present inside a `MultiSeqError::
    /// CapabilityUnsupported { capability: "...xlen..." }` constructor
    /// call for the DEFAULT path — the typed error is GONE except
    /// for the defense-in-depth mixed-presence invariant violation.
    #[test]
    fn h189_iter2b_xlen_hybrid_branch_typed_error_replaced_with_slot_routing() {
        let src = include_str!("../forward_prefill.rs");
        let fn_marker = "pub fn forward_prefill_with_soft_tokens_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H189: prefill slot-aware fn marker present (H84 asserts)");
        let fn_window = &src[fn_idx..(fn_idx + 80_000).min(src.len())];

        // (a) The iter-2B-xlen typed-deferral capability literal — EXACT
        // string a CapabilityUnsupported constructor would have used at
        // iter-2B SHIP.  iter-2B-xlen REMOVES it from the default path.
        let iter2b_xlen_typed_error_default =
            "gemma4-forward-prefill-slot-N-hybrid-xlen (iter-B4c-kernel-iter-2B-xlen per ADR-040 §6.1.34";
        assert!(
            !fn_window.contains(iter2b_xlen_typed_error_default),
            "H189 FALSIFIED: prefill fn body still contains the iter-2B \
             xlen typed-error capability label `{iter2b_xlen_typed_error_default}` \
             — iter-2B-xlen slot routing NOT landed; \
             HF2Q_DFLASH_XLEN_SDPA=1 requests still surface \
             CapabilityUnsupported at the hybrid xlen branch."
        );

        // (b) Positive pin: `xlen_engaged` binding present (the
        // load-bearing predicate for the conditional materialization).
        assert!(
            fn_window.contains("let xlen_engaged ="),
            "H189 FALSIFIED: prefill fn body does NOT contain the \
             `let xlen_engaged =` binding — iter-2B-xlen slot routing \
             primitive missing."
        );

        // (c) Positive pin: `bf16_xlen_k: bf16_xlen_k_view,` non-None
        // propagation through the `HybridKvBuffers {}` constructor.
        // PRE-iter-2B-xlen state had hardcoded `bf16_xlen_k: None,
        // bf16_xlen_v: None,` literals.  iter-2B-xlen REPLACES them
        // with the conditional materialization output.
        assert!(
            fn_window.contains("bf16_xlen_k: bf16_xlen_k_view,"),
            "H189 FALSIFIED: prefill fn body does NOT contain the \
             `bf16_xlen_k: bf16_xlen_k_view,` non-None struct-field \
             binding — iter-2B-xlen propagation NOT wired into the \
             `HybridKvBuffers {{}}` constructor."
        );
        assert!(
            fn_window.contains("bf16_xlen_v: bf16_xlen_v_view,"),
            "H189 FALSIFIED: prefill fn body does NOT contain the \
             `bf16_xlen_v: bf16_xlen_v_view,` non-None struct-field \
             binding — iter-2B-xlen V-side propagation NOT wired."
        );

        // (d) The iter-2B-xlen label substring IS preserved somewhere
        // in the fn body as doc-comment cite (operator-grep'able
        // forward pointer — required by H87 transitivity).
        assert!(
            fn_window.contains("iter-B4c-kernel-iter-2B-xlen per ADR-040 §6.1.47"),
            "H189 FALSIFIED: prefill fn body does NOT contain the \
             operator-grep'able iter-2B-xlen forward pointer \
             `iter-B4c-kernel-iter-2B-xlen per ADR-040 §6.1.47`."
        );
    }

    /// **H190 (skip-mode)** — iter-2-decode-A-xlen hybrid xlen
    /// branch typed-error REPLACED with real slot routing.
    ///
    /// Mirror of H189 applied to `forward_decode_slot_aware`'s
    /// hybrid branch.  Same 4 pins (typed-error removed, xlen_engaged
    /// binding present, struct-field bindings present, doc-cite
    /// preserved) but searched inside the decode fn body window.
    #[test]
    fn h190_iter2_decode_a_xlen_hybrid_branch_typed_error_replaced_with_slot_routing() {
        let src = include_str!("../forward_prefill.rs");
        let fn_marker = "pub fn forward_decode_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H190: decode slot-aware fn marker present (H123 asserts)");
        let fn_window = &src[fn_idx..(fn_idx + 80_000).min(src.len())];

        // (a) Decode-side iter-2-decode-A-xlen typed-deferral capability
        // literal REMOVED from the default path.
        let iter2_decode_xlen_typed_error_default =
            "gemma4-forward-decode-slot-N-hybrid-xlen (iter-B4c-kernel-iter-2-decode-A-xlen per ADR-040 §6.1.38";
        assert!(
            !fn_window.contains(iter2_decode_xlen_typed_error_default),
            "H190 FALSIFIED: decode fn body still contains the \
             iter-2-decode-A-xlen typed-error capability label \
             `{iter2_decode_xlen_typed_error_default}` — \
             iter-2-decode-A-xlen slot routing NOT landed."
        );

        // (b) Positive pin: `xlen_engaged` binding present in decode body.
        assert!(
            fn_window.contains("let xlen_engaged ="),
            "H190 FALSIFIED: decode fn body does NOT contain the \
             `let xlen_engaged =` binding — iter-2-decode-A-xlen slot \
             routing primitive missing on the decode side."
        );

        // (c) Positive pin: `bf16_xlen_k: bf16_xlen_k_view,` non-None
        // propagation in the decode body's `HybridKvBuffers {}` struct.
        assert!(
            fn_window.contains("bf16_xlen_k: bf16_xlen_k_view,"),
            "H190 FALSIFIED: decode fn body does NOT contain the \
             `bf16_xlen_k: bf16_xlen_k_view,` non-None struct-field \
             binding — iter-2-decode-A-xlen propagation NOT wired."
        );
        assert!(
            fn_window.contains("bf16_xlen_v: bf16_xlen_v_view,"),
            "H190 FALSIFIED: decode fn body does NOT contain the \
             `bf16_xlen_v: bf16_xlen_v_view,` non-None struct-field \
             binding — iter-2-decode-A-xlen V-side propagation NOT wired."
        );

        // (d) Decode-side iter-2-decode-A-xlen forward-pointer cite.
        assert!(
            fn_window.contains("iter-B4c-kernel-iter-2-decode-A-xlen per ADR-040 §6.1.47"),
            "H190 FALSIFIED: decode fn body does NOT contain the \
             operator-grep'able iter-2-decode-A-xlen forward pointer \
             `iter-B4c-kernel-iter-2-decode-A-xlen per ADR-040 §6.1.47`."
        );
    }

    /// **H191 (skip-mode)** — per-slot byte offset arithmetic uses
    /// `* 2u64 // BF16 = 2 bytes/elem` AND references
    /// `k_elems_per_slot` (the F16 K stride formula — BF16 K shape is
    /// identical to F16 K so the elem-count formula reuses).
    ///
    /// Defends against:
    ///   * Accidentally using `* 4u64` (F32 stride) on the BF16
    ///     buffer (would compute 2x the true offset → slot N writes
    ///     to slot 2N's region → cross-slot interference).
    ///   * Accidentally using `* 1u64` (U8 stride) (would compute
    ///     1/2x the true offset → silent corruption of mid-slot
    ///     bytes).
    ///   * Forgetting to multiply by slot_id.0 (would route every
    ///     slot to slot 0's xlen region).
    #[test]
    fn h191_xlen_byte_offset_per_bf16_layout() {
        let src = include_str!("../forward_prefill.rs");
        let fn_marker = "pub fn forward_prefill_with_soft_tokens_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H191: prefill slot-aware fn marker present");
        let fn_window = &src[fn_idx..(fn_idx + 80_000).min(src.len())];

        // (a) BF16 = 2 bytes/elem comment + literal multiplier present
        // in the xlen byte computation.  Pin the exact comment substring
        // so a regression to F32/U8 stride would falsify.
        assert!(
            fn_window.contains("checked_mul(2u64) // BF16 = 2 bytes/elem"),
            "H191 FALSIFIED: prefill fn body does NOT contain the \
             load-bearing `checked_mul(2u64) // BF16 = 2 bytes/elem` \
             literal — iter-2B-xlen byte-offset arithmetic at risk \
             of wrong-stride regression."
        );

        // (b) `xlen_bytes_per_slot` binding derived from
        // `k_elems_per_slot` (the F16 K formula — BF16 reuses).
        assert!(
            fn_window.contains("let xlen_bytes_per_slot: u64 = (k_elems_per_slot as u64)"),
            "H191 FALSIFIED: prefill fn body does NOT derive \
             `xlen_bytes_per_slot` from `k_elems_per_slot` — BF16 \
             stride reuses the F16 K elem count formula (identical \
             `[nkv, cap, hd]` shape per the alloc helper); \
             regression risk: future iter accidentally re-derives \
             elem count with wrong shape factor."
        );

        // (c) Mirror checks for the decode body.
        let decode_marker = "pub fn forward_decode_slot_aware(";
        let decode_idx = src
            .find(decode_marker)
            .expect("H191: decode slot-aware fn marker present");
        let decode_window = &src[decode_idx..(decode_idx + 80_000).min(src.len())];
        assert!(
            decode_window.contains("checked_mul(2u64) // BF16 = 2 bytes/elem"),
            "H191 FALSIFIED: decode fn body does NOT contain the \
             `checked_mul(2u64) // BF16 = 2 bytes/elem` literal — \
             iter-2-decode-A-xlen byte-offset arithmetic at risk."
        );
        assert!(
            decode_window.contains("let xlen_bytes_per_slot: u64 = (k_elems_per_slot as u64)"),
            "H191 FALSIFIED: decode fn body does NOT derive \
             `xlen_bytes_per_slot` from `k_elems_per_slot`."
        );
    }

    /// **H192 (skip-mode)** — per-slot byte isolation: the
    /// `xlen_byte_offset` binding uses `slot_id.0` as the multiplier
    /// on `xlen_bytes_per_slot`, not a hardcoded 0.
    ///
    /// Defends against a defective copy-paste from the F16 K
    /// computation that accidentally hardcodes `0u64.checked_mul(...)`
    /// or omits the `slot_id.0` factor entirely — which would route
    /// every SlotId(N>0) request's xlen K + V writes to slot 0's
    /// byte region, causing silent cross-slot corruption.
    ///
    /// Slot 0's xlen byte offset is `0` by arithmetic (`0 * stride
    /// == 0`); slot N's xlen byte offset is `N * stride`.  Per-slot
    /// byte isolation is enforced at the slice_view layer (Metal
    /// `setBuffer:offset:atIndex:` semantics route the byte offset
    /// into kernel dispatch — same as the iter-2B F16 K + V slot
    /// isolation).
    #[test]
    fn h192_per_slot_xlen_byte_isolation() {
        let src = include_str!("../forward_prefill.rs");

        // (a) Prefill side: `xlen_byte_offset` derivation uses
        // `(slot_id.0 as u64).checked_mul(xlen_bytes_per_slot)` —
        // pinned by the exact substring.
        let prefill_marker = "pub fn forward_prefill_with_soft_tokens_slot_aware(";
        let prefill_idx = src
            .find(prefill_marker)
            .expect("H192: prefill slot-aware fn marker present");
        let prefill_window = &src[prefill_idx..(prefill_idx + 80_000).min(src.len())];
        assert!(
            prefill_window.contains("let xlen_byte_offset: u64 = (slot_id.0 as u64)",),
            "H192 FALSIFIED: prefill fn body does NOT contain the \
             `let xlen_byte_offset: u64 = (slot_id.0 as u64)` binding \
             — per-slot byte isolation at risk; SlotId(N>0) xlen \
             writes would target slot 0's region if slot_id.0 is \
             omitted from the byte-offset multiplication."
        );
        assert!(
            prefill_window.contains(".checked_mul(xlen_bytes_per_slot)"),
            "H192 FALSIFIED: prefill fn body does NOT chain \
             `.checked_mul(xlen_bytes_per_slot)` on the slot_id.0 \
             factor — byte-offset overflow guard at risk."
        );

        // (b) Decode side: same pins.
        let decode_marker = "pub fn forward_decode_slot_aware(";
        let decode_idx = src
            .find(decode_marker)
            .expect("H192: decode slot-aware fn marker present");
        let decode_window = &src[decode_idx..(decode_idx + 80_000).min(src.len())];
        assert!(
            decode_window.contains("let xlen_byte_offset: u64 = (slot_id.0 as u64)",),
            "H192 FALSIFIED: decode fn body does NOT contain the \
             `let xlen_byte_offset: u64 = (slot_id.0 as u64)` binding \
             — decode-side per-slot byte isolation at risk."
        );
    }

    /// **H193 (skip-mode)** — default OFF (HF2Q_DFLASH_XLEN_SDPA
    /// unset) path UNCHANGED.
    ///
    /// When `xlen_engaged == false`, the conditional materialization
    /// produces `(None, None)` and the `HybridKvBuffers {}`
    /// constructor receives `bf16_xlen_k: None, bf16_xlen_v: None`
    /// equivalent to PRE-iter-2B-xlen + iter-2-decode-A-xlen
    /// behavior.
    ///
    /// Pinned by source-grep on the `else` arm of the materialization
    /// that produces `(None, None)`.  Defends against a regression
    /// that accidentally allocates fresh xlen buffers per call on
    /// the default-OFF path (which would 5-7x the per-call alloc
    /// overhead + violate the alloc-time decision discipline).
    #[test]
    fn h193_default_xlen_off_path_unchanged() {
        let src = include_str!("../forward_prefill.rs");

        // (a) Prefill body: `else { (None, None) }` materialization
        // present (the default-OFF fall-through).
        let prefill_marker = "pub fn forward_prefill_with_soft_tokens_slot_aware(";
        let prefill_idx = src
            .find(prefill_marker)
            .expect("H193: prefill slot-aware fn marker present");
        let prefill_window = &src[prefill_idx..(prefill_idx + 80_000).min(src.len())];

        // Conservative grep: the `(None, None)` literal appears in the
        // materialization's else arm — at least once on the prefill
        // side AND at least once on the decode side.  Counting both:
        // 2 occurrences across the whole fn-region (one per fn).
        let prefill_none_none_count = prefill_window.matches("(None, None)").count();
        assert!(
            prefill_none_none_count >= 1,
            "H193 FALSIFIED: prefill fn body does NOT contain at \
             least one `(None, None)` materialization fall-through \
             — default OFF path may now over-allocate xlen buffers."
        );

        // (b) Decode body: same.
        let decode_marker = "pub fn forward_decode_slot_aware(";
        let decode_idx = src
            .find(decode_marker)
            .expect("H193: decode slot-aware fn marker present");
        let decode_window = &src[decode_idx..(decode_idx + 80_000).min(src.len())];
        let decode_none_none_count = decode_window.matches("(None, None)").count();
        assert!(
            decode_none_none_count >= 1,
            "H193 FALSIFIED: decode fn body does NOT contain at \
             least one `(None, None)` materialization fall-through \
             — default OFF path may now over-allocate xlen buffers \
             on the decode side."
        );

        // (c) The `xlen_engaged` predicate is bound via `.any(...)`
        // on the buffer fields — confirms the predicate detects
        // alloc-time presence, not env-var reading.  Defends against
        // a regression that reads `std::env::var("HF2Q_DFLASH_XLEN_SDPA")`
        // at slot-routing time (which would diverge from the alloc-
        // time decision per the LazyLock-cache discipline).
        assert!(
            prefill_window
                .contains(".any(|buf| buf.bf16_xlen_k.is_some() || buf.bf16_xlen_v.is_some())",),
            "H193 FALSIFIED: prefill fn body does NOT derive \
             `xlen_engaged` from buffer-field presence — risk of \
             slot-routing-time env-var read diverging from alloc-time \
             decision."
        );
    }

    /// **H194 (skip-mode)** — SerialFifo + HF2Q_DFLASH_XLEN_SDPA=1
    /// byte equivalence preserved at SlotId(0).
    ///
    /// Mirror of H86 + H102 + H128 byte-equivalence pin chain
    /// extended to the xlen surface:
    ///
    ///   * Sibling fn `forward_prefill_with_soft_tokens_resume`
    ///     signature UNCHANGED — no new xlen-specific params (the
    ///     xlen buffers are consumed via `self.hybrid_kv` strong-
    ///     ref, same as the F16 K + V).
    ///   * Sibling fn `forward_decode` signature UNCHANGED.
    ///   * The iter-2B-xlen + iter-2-decode-A-xlen routing materializes
    ///     slot-views with byte offset 0 at SlotId(0); slice_view(0,
    ///     n_elements) produces a view byte-identical to the original
    ///     buffer.  Combined with code-path disjointness (iter-1
    ///     worker-arm predicate gates this fn on SlotId(N>0)),
    ///     SerialFifo never reaches this fn AT ALL — the byte
    ///     equivalence pin is preserved by routing exclusion, not
    ///     by routing identity.
    #[test]
    fn h194_serial_fifo_xlen_byte_equivalence_preserved() {
        let src = include_str!("../forward_prefill.rs");

        // (a) Sibling fn forward_prefill_with_soft_tokens_resume
        // signature UNCHANGED (no slot_id / multi_seq_kv / xlen params
        // — H86 transitivity to xlen).
        let sibling_marker = "pub fn forward_prefill_with_soft_tokens_resume(";
        let sibling_idx = src
            .find(sibling_marker)
            .expect("H194: sibling fn forward_prefill_with_soft_tokens_resume present");
        let sibling_sig = &src[sibling_idx..(sibling_idx + 4_000).min(src.len())];
        assert!(
            !sibling_sig.contains("slot_id:"),
            "H194 FALSIFIED: sibling `forward_prefill_with_soft_tokens_resume` \
             signature contains `slot_id:` — H86 byte-equivalence pin \
             broken at xlen layer."
        );
        assert!(
            !sibling_sig.contains("multi_seq_kv"),
            "H194 FALSIFIED: sibling fn signature contains \
             `multi_seq_kv` — H86 byte-equivalence pin broken."
        );
        assert!(
            !sibling_sig.contains("bf16_xlen"),
            "H194 FALSIFIED: sibling fn signature contains \
             `bf16_xlen` — iter-2B-xlen accidentally exposed an xlen \
             parameter on the SerialFifo-routing sibling, breaking \
             H86 byte-equivalence."
        );

        // (b) iter-2B-xlen materialization is INSIDE the slot-aware
        // fn body (not in the sibling).  The iter-2B prefill mount
        // assignment `self.hybrid_kv = Some(slot_view_hybrid);`
        // STILL present (transitivity to H101).
        assert!(
            src.contains("self.hybrid_kv = Some(slot_view_hybrid)"),
            "H194 FALSIFIED: iter-2B mount `self.hybrid_kv = \
             Some(slot_view_hybrid)` REGRESSED — iter-2B-xlen \
             accidentally broke H101."
        );
    }

    /// **H195 (skip-mode)** — production-default non-xlen + HB-encoded
    /// + dense F32 + legacy 4-bit surfaces UNCHANGED.
    ///
    /// Composite transitivity pin from H97 (iter-2B production-default
    /// hybrid branch landed) + H174 (iter-2A-cont HB-encoded prefill)
    /// + H175 (iter-2-decode-B HB-encoded decode) + H123
    /// (forward_decode_slot_aware fn defined) + H181 (iter-2C 4-bit
    /// prefill) + H182 (iter-2D dense F32 prefill).  Defends against
    /// any iter-2B-xlen + iter-2-decode-A-xlen body insertion that
    /// accidentally regresses one of those production surfaces.
    ///
    /// Also pins:
    ///   * Qwen35 + Qwen3VL slot-aware orchestrators UNCHANGED.
    ///   * ADR-040 §6.1.47 closure block exists (forward-pointer
    ///     destination for the iter-2B-xlen + iter-2-decode-A-xlen
    ///     joint deferral cites).
    #[test]
    fn h195_production_default_and_qwen35_qwen3vl_surfaces_unchanged() {
        let pf_src = include_str!("../forward_prefill.rs");

        // (a) iter-2B hybrid-branch typed-error label STILL REMOVED
        // (H97 transitivity).
        let iter2a_hybrid_typed_error =
            "gemma4-forward-prefill-slot-N-hybrid (iter-B4c-kernel-iter-2B per";
        assert!(
            !pf_src.contains(iter2a_hybrid_typed_error),
            "H195 FALSIFIED: iter-2A hybrid-branch typed-error label \
             REGRESSED — iter-2B-xlen accidentally restored the iter-2A \
             typed-error on the production-default hybrid branch."
        );

        // (b) iter-2B mount STILL present (H101 transitivity).
        assert!(
            pf_src.contains("self.hybrid_kv = Some(slot_view_hybrid)"),
            "H195 FALSIFIED: iter-2B mount REGRESSED."
        );

        // (c) iter-2-decode-A fn STILL defined (H123 transitivity).
        assert!(
            pf_src.contains("pub fn forward_decode_slot_aware("),
            "H195 FALSIFIED: `forward_decode_slot_aware` fn removed — \
             H123 transitivity broken."
        );

        // (d) iter-2A-cont + iter-2-decode-B HB-encoded mount STILL
        // present (H174 + H175 transitivity).
        assert!(
            pf_src.matches("self.leg_hb_encoded = Some(").count() >= 2,
            "H195 FALSIFIED: `self.leg_hb_encoded = Some(` mount count \
             < 2 (expected ≥2: one in prefill, one in decode).  \
             iter-2A-cont (H174) or iter-2-decode-B (H175) regression."
        );

        // (e) iter-2C 4-bit (Vec-swap) mount STILL present
        // (H181 + H183 transitivity).
        assert!(
            pf_src
                .matches("std::mem::replace(&mut self.kv_caches,")
                .count()
                >= 2,
            "H195 FALSIFIED: 4-bit `std::mem::replace(&mut self.kv_caches,` \
             mount count < 2 (expected ≥2: prefill + decode) — \
             iter-2C (H181) or iter-2-decode-D-4bit (H183) regression."
        );

        // (f) iter-2D dense F32 mount STILL present
        // (H182 + H184 transitivity).
        assert!(
            pf_src.contains("self.dense_kvs = Some(slot_view_dense)"),
            "H195 FALSIFIED: iter-2D dense F32 mount \
             `self.dense_kvs = Some(slot_view_dense)` REGRESSED."
        );

        let src = include_str!("./engine.rs");
        // (g) Qwen35 worker-arm slot-aware fns UNCHANGED.
        for required in [
            "generate_qwen35_once_slot_aware",
            "generate_stream_qwen35_once_slot_aware",
        ] {
            assert!(
                src.contains(required),
                "H195 FALSIFIED: Qwen35 slot-aware surface `{required}` \
                 removed — iter-2B-xlen accidentally touched Qwen35 \
                 architecture."
            );
        }

        // (h) Embed-arm fn body does NOT call forward_decode_slot_aware
        // (mirror of H180 / H188).
        let fn_marker = "fn embed_gemma4_slot_aware(";
        let embed_idx = src
            .find(fn_marker)
            .expect("H195: embed_gemma4_slot_aware not found");
        let embed_window = &src[embed_idx..(embed_idx + 10_000).min(src.len())];
        assert!(
            !embed_window.contains(".forward_decode_slot_aware("),
            "H195 FALSIFIED: Embed-arm fn body calls \
             `forward_decode_slot_aware`. The Embed-arm has NO decode \
             loop — calling forward_decode_slot_aware would corrupt the \
             L2-normalized embedding vector."
        );

        // (i) ADR-040 §6.1.47 closure block exists.
        let adr =
            crate::serve::api::engine::adr040_history_doc() /* iter-230 A1: §6.1.x moved to history (aeb6e87c) */;
        assert!(
            adr.contains("### 6.1.47"),
            "H195 FALSIFIED: ADR-040 §6.1.47 closure block not found. \
             iter-2B-xlen + iter-2-decode-A-xlen sub-deferral cites \
             point at a non-existent destination."
        );
    }
}

// ════════════════════════════════════════════════════════════════════════════
// ADR-040 Phase B iter-B4c-kernel iter-2-decode-C-stream-tool-call
// (Gemma 4 GenerateStream-arm slot-aware streaming tool-call body
// emission via Wave 3 W-B3 ToolCallStreamEmitter) — 2026-05-30
// ────────────────────────────────────────────────────────────────────────────
//
// Closes the surviving iter-2-decode-C sub-deferral pinned at §6.1.39:
// **streaming tool-call body emission**.  The slot-aware streaming
// orchestrator `generate_stream_gemma4_once_slot_aware` previously
// entry-checked a `stream_tool_call_engaged` predicate and surfaced
// a typed `MultiSeqError::CapabilityUnsupported` SSE Error event
// when a `ToolCallSplitter` was registered for the model AND
// `grammar_kind ∈ {ToolCallBodyAuto, ToolCallBodyRequired}`.  This
// iter REPLACES that short-circuit with the real Wave 3 W-B3
// `ToolCallStreamEmitter` plumbing: per-fragment `advance(body,
// events)` + per-call `finalize(body, reg, policy, tc_index, saw_tc,
// events)`, mirror of `generate_stream_once`'s `route_content`
// closure at engine.rs:12210-12317.
//
// Tests (H196–H201):
//
//   H196 (skip-mode): GenerateStream-arm typed-error literal for the
//                     `stream_tool_call_engaged` short-circuit REMOVED
//                     from the fn body.  Positive pin: the body now
//                     constructs `ToolCallStreamEmitter::new(reg.map(
//                     |r| r.family), *tc_index)` at ToolCallOpen, and
//                     calls `em.advance(body, event_sink)` on
//                     ToolCallText + `em.finalize(...)` on
//                     ToolCallClose — verbatim mirror of the non-slot-
//                     aware `route_content` shape.
//   H197 (skip-mode): per-fragment incremental JSON streaming wired.
//                     Source-grep confirms `em.advance(body,` AND
//                     `em.finalize(` BOTH appear inside the slot-aware
//                     fn body (NOT just in the non-slot-aware sibling
//                     at engine.rs:12278-12312).
//   H198 (skip-mode): SerialFifo + SlotId(0) byte-equivalence
//                     preserved via H135 transitivity — the sibling
//                     `forward_decode` signature in gemma4/forward_gpu
//                     .rs is STILL unchanged (no slot_id /
//                     multi_seq_kv params); iter-2-decode-A's
//                     `forward_decode_slot_aware` signature is STILL
//                     unchanged; code-path disjointness at the
//                     worker-arm `slot_id != SlotId(0)` predicate
//                     short-circuits the slot-aware orchestrator for
//                     SerialFifo + SlotId(0) routes.
//   H199 (skip-mode): Qwen35 iter-2 GenerateStream-arm slot-aware
//                     streaming tool-call surface UNCHANGED.  The
//                     Qwen35 fn `generate_stream_qwen35_once_extended_
//                     slot_aware` body is NOT touched.  The Qwen35
//                     architecture's tool-call streaming was already
//                     handled at iter-C2d-cont-kernel iter-2 §6.1.28
//                     via a different surface area; iter-2-decode-C-
//                     stream-tool-call does NOT regress it.
//   H200 (skip-mode): SSE event ordering pin.  The Gemma 4 streaming
//                     fn body, after iter-2-decode-C-stream-tool-call,
//                     emits a terminal `Done { finish_reason, .. }`
//                     event at the end of every successful decode
//                     path; the `finish_reason` is overridden to
//                     `"tool_calls"` when `saw_tool_call` latched true
//                     during the decode loop (matches the OpenAI
//                     tool-calls finish_reason spec).  No new SSE
//                     terminal Error event is added on the happy path
//                     (the iter-2-decode-C-stream-tool-call typed-
//                     error abort is REMOVED).
//   H201 (skip-mode): orthogonal surfaces UNCHANGED.  Embed-arm fn
//                     body does NOT call `forward_decode_slot_aware`
//                     (H136 transitivity).  Qwen3VL forward paths
//                     UNCHANGED.  Surviving sub-deferral label
//                     `iter-B4c-kernel-iter-2-decode-C-stream-tool-
//                     call per ADR-040 §6.1.39` is STILL grep-able
//                     in engine.rs as a doc-comment cite (the label
//                     substring is preserved per H87 forward-pointer
//                     discoverability discipline).  ADR-040 §6.1.48
//                     closure block exists (forward-pointer dest).

#[cfg(test)]
mod adr040_phase_b_iter_b4c_kernel_iter2_decode_c_stream_tool_call_gemma4_tests {
    // Skip-mode source-grep tests; intentionally NO `use super::*;`.

    /// **H196 (skip-mode)** — iter-2-decode-C-stream-tool-call typed-
    /// error short-circuit REPLACED with real Wave 3 W-B3
    /// `ToolCallStreamEmitter` plumbing.
    ///
    /// (a) The iter-2-decode-C typed-deferral capability literal
    ///     `gemma4-forward-decode-stream-slot-N-tool-call-body
    ///     (iter-B4c-kernel-iter-2-decode-C-stream-tool-call per
    ///     ADR-040 §6.1.39 — streaming tool-call body emission via
    ///     Wave 3 W-B3 ToolCallStreamEmitter` is REMOVED from the
    ///     slot-aware streaming fn body — the typed `MultiSeqError::
    ///     CapabilityUnsupported` constructor that pre-iter-2-decode-
    ///     C-stream-tool-call branched on `stream_tool_call_engaged`
    ///     is GONE.
    /// (b) Positive pin: `ToolCallStreamEmitter::new(` appears in the
    ///     slot-aware streaming fn body (per-call emitter
    ///     construction at ToolCallOpen).
    /// (c) Positive pin: `tool_call_policy = params.tool_call_policy`
    ///     binding present (the policy passthrough to `finalize`'s
    ///     fallback dispatch).
    ///
    /// Note: the iter-2-decode-C-stream-tool-call label substring is
    /// PRESERVED somewhere in the fn body as a doc-comment cite (H87
    /// forward-pointer discoverability — see H201).  H196 only pins
    /// the typed-error constructor + its specific capability literal
    /// (which carries the load-bearing "is out of iter-2-decode-C
    /// scope" phrasing) are GONE.
    #[test]
    fn h196_stream_tool_call_typed_error_replaced_with_real_emitter() {
        let src = include_str!("engine.rs");
        let fn_marker = "fn generate_stream_gemma4_once_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H196: generate_stream_gemma4_once_slot_aware not found");
        // Window covers the prefill-Ok branch + entire decode loop +
        // terminal Done emission.  Body grew to ~33k bytes post-
        // iter-2-decode-C-stream-tool-call (~3k delta for inner
        // route_content closure + tool-call state vars).
        let fn_window = &src[fn_idx..(fn_idx + 60_000).min(src.len())];

        // (a) The exact iter-2-decode-C typed-error capability literal
        // — its presence INSIDE a `MultiSeqError::CapabilityUnsupported
        // { capability: ... }` constructor was the load-bearing typed
        // deferral for the streaming tool-call body.  iter-2-decode-C-
        // stream-tool-call REMOVES that constructor + capability
        // literal pairing.
        let iter2_decode_c_tc_capability =
            "gemma4-forward-decode-stream-slot-N-tool-call-body (iter-B4c-kernel-iter-2-decode-C-stream-tool-call per ADR-040 §6.1.39 — streaming tool-call body emission";
        assert!(
            !fn_window.contains(iter2_decode_c_tc_capability),
            "H196 FALSIFIED: slot-aware streaming fn body still \
             contains the iter-2-decode-C-stream-tool-call typed-error \
             capability literal `{iter2_decode_c_tc_capability}` — \
             ToolCallStreamEmitter plumbing NOT landed; streaming \
             tool-call requests at SlotId(N>0) still surface a typed \
             CapabilityUnsupported SSE Error event."
        );

        // (b) Positive pin: per-call emitter construction at
        // ToolCallOpen.  The body MUST construct a fresh
        // ToolCallStreamEmitter per call, mirror of the non-slot-aware
        // route_content at engine.rs:12257.
        assert!(
            fn_window.contains("ToolCallStreamEmitter::new("),
            "H196 FALSIFIED: slot-aware streaming fn body does NOT \
             construct `ToolCallStreamEmitter::new(...)` — Wave 3 \
             W-B3 incremental tool-call streaming NOT wired."
        );

        // (c) Positive pin: tool_call_policy passthrough.  The closure
        // must capture `tool_call_policy = params.tool_call_policy` so
        // ToolCallStreamEmitter::finalize's fallback (close-buffered)
        // dispatch enforces the Constrained-vs-Auto loud-error policy.
        assert!(
            fn_window.contains("let tool_call_policy = params.tool_call_policy"),
            "H196 FALSIFIED: slot-aware streaming fn body does NOT \
             bind `tool_call_policy = params.tool_call_policy` — \
             ToolCallStreamEmitter::finalize cannot enforce the \
             Constrained-vs-Auto policy branch on parse failure."
        );
    }

    /// **H197 (skip-mode)** — per-fragment incremental JSON streaming
    /// wired via `em.advance(...)` + `em.finalize(...)` inside the
    /// slot-aware streaming fn body.
    ///
    /// (a) `em.advance(body, event_sink)` present — drives the
    ///     incremental name + kv-pair emission on each ToolCallText
    ///     fragment.
    /// (b) `em.finalize(` present — emits the closing `}` + any tail
    ///     kvs at ToolCallClose.
    /// (c) `saw_tool_call` latch present — the finish_reason override
    ///     to `"tool_calls"` requires this latch be readable at end-of-
    ///     decode.
    #[test]
    fn h197_per_fragment_incremental_streaming_wired_in_slot_aware_fn() {
        let src = include_str!("engine.rs");
        let fn_marker = "fn generate_stream_gemma4_once_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H197: generate_stream_gemma4_once_slot_aware not found");
        let fn_window = &src[fn_idx..(fn_idx + 60_000).min(src.len())];

        // (a) advance call on the per-call emitter, INSIDE the slot-
        // aware fn body — not just in the non-slot-aware sibling.
        assert!(
            fn_window.contains("em.advance(body, event_sink)"),
            "H197 FALSIFIED: slot-aware streaming fn body does NOT \
             call `em.advance(body, event_sink)` — Wave 3 W-B3 \
             per-fragment incremental tool-call argument streaming \
             NOT wired in the slot-aware path."
        );

        // (b) finalize call on the per-call emitter at ToolCallClose.
        assert!(
            fn_window.contains("em.finalize("),
            "H197 FALSIFIED: slot-aware streaming fn body does NOT \
             call `em.finalize(...)` — Wave 3 W-B3 finalize-on-close \
             dispatch NOT wired."
        );

        // (c) saw_tool_call latch present (drives finish_reason
        // override per OpenAI tool-calls spec).
        assert!(
            fn_window.contains("saw_tool_call"),
            "H197 FALSIFIED: slot-aware streaming fn body does NOT \
             reference `saw_tool_call` — finish_reason override to \
             `\"tool_calls\"` at end-of-decode NOT wired (OpenAI \
             spec violation)."
        );

        // (d) finish_reason override to `\"tool_calls\"` when
        // saw_tool_call latches.  Pin the EXACT shape so a regression
        // to the iter-2-decode-C-only `\"stop\"`/`\"length\"`-only
        // finish path falsifies.
        assert!(
            fn_window.contains("finish_reason = \"tool_calls\""),
            "H197 FALSIFIED: slot-aware streaming fn body does NOT \
             override `finish_reason = \"tool_calls\"` on the \
             saw_tool_call latch — OpenAI tool-calls finish_reason \
             contract broken at SlotId(N>0)."
        );
    }

    /// **H198 (skip-mode)** — SerialFifo + SlotId(0) byte-equivalence
    /// preserved via H135 transitivity.  iter-2-decode-C-stream-tool-
    /// call touches ONLY the orchestrator body — sibling fn signatures
    /// in `gemma4/forward_gpu.rs` (forward_decode) AND in
    /// `serve/forward_prefill.rs` (forward_decode_slot_aware) are
    /// UNCHANGED (additive-zero).
    #[test]
    fn h198_serial_fifo_sibling_forward_decode_signature_unchanged() {
        // (a) Sibling forward_decode signature unchanged (mirror of
        // H135).
        let src = include_str!("../../inference/models/gemma4/forward_gpu.rs");
        let sibling_marker = "pub fn forward_decode(";
        let sib_idx = src
            .find(sibling_marker)
            .expect("H198: sibling forward_decode signature missing");
        let sig_end = src[sib_idx..]
            .find(") -> Result<u32>")
            .map(|off| sib_idx + off + ") -> Result<u32>".len())
            .unwrap_or(sib_idx + 600);
        let sig_window = &src[sib_idx..sig_end.min(src.len())];
        assert!(
            !sig_window.contains("slot_id"),
            "H198 FALSIFIED: sibling `forward_decode` signature \
             contains `slot_id`. iter-2-decode-C-stream-tool-call \
             discipline broken — sibling fn signature MUST remain \
             unchanged."
        );
        assert!(
            !sig_window.contains("multi_seq_kv"),
            "H198 FALSIFIED: sibling `forward_decode` signature \
             mentions `multi_seq_kv`. iter-2-decode-C-stream-tool-call \
             discipline broken — SerialFifo decode path MUST NOT \
             consume the multi-seq scaffold."
        );

        // (b) iter-2-decode-A's forward_decode_slot_aware signature
        // STILL present (iter-2-decode-C-stream-tool-call is additive
        // to the orchestrator body, not to the model fn).
        let pf_src = include_str!("../forward_prefill.rs");
        assert!(
            pf_src.contains("pub fn forward_decode_slot_aware("),
            "H198 FALSIFIED: iter-2-decode-A's `forward_decode_slot_\
             aware` signature is missing from forward_prefill.rs. \
             iter-2-decode-C-stream-tool-call accidentally removed \
             the load-bearing primitive."
        );
    }

    /// **H199 (skip-mode)** — Qwen35 + Qwen3VL UNCHANGED.  The Qwen35
    /// streaming slot-aware fn `generate_stream_qwen35_once_extended_
    /// slot_aware` body is NOT touched.
    #[test]
    fn h199_qwen35_and_qwen3vl_surfaces_unchanged() {
        let src = include_str!("engine.rs");
        // Qwen35 slot-aware fns STILL defined (mirror of H136).
        for required in [
            "generate_qwen35_once_slot_aware(",
            "generate_stream_qwen35_once_extended_slot_aware(",
            "embed_qwen35_slot_aware(",
            "generate_qwen35_once_with_soft_tokens_slot_aware(",
        ] {
            assert!(
                src.contains(required),
                "H199 FALSIFIED: Qwen35 slot-aware fn `{required}` \
                 is NOT present — iter-2-decode-C-stream-tool-call \
                 accidentally regressed a Qwen35 lift."
            );
        }

        // Qwen35 source files NOT touched by this iter — pin via
        // gpu_delta_net.rs surface (A2b-cont landing).  The token
        // `iter-2-decode-C-stream-tool-call` MUST NOT appear in the
        // Qwen35 architecture source (iter-2-decode-C-stream-tool-
        // call is a Gemma-only label).
        let qwen35_src = include_str!("../../inference/models/qwen35/gpu_delta_net.rs");
        assert!(
            !qwen35_src.contains("iter-2-decode-C-stream-tool-call"),
            "H199 FALSIFIED: Qwen35 gpu_delta_net.rs mentions \
             `iter-2-decode-C-stream-tool-call`. The iter scope is \
             Gemma 4 GenerateStream-arm only — Qwen35 architecture \
             accidentally touched."
        );
    }

    /// **H200 (skip-mode)** — SSE event ordering pin.  The slot-aware
    /// streaming fn body emits a terminal `Done { finish_reason, ..
    /// }` event at end-of-decode AND the `finish_reason` is
    /// overridden to `"tool_calls"` when saw_tool_call latched.
    /// Defends against:
    ///   * Accidentally emitting a terminal SSE Error event on the
    ///     happy path (the iter-2-decode-C typed-error abort is
    ///     REMOVED).
    ///   * Forgetting to override finish_reason (would surface as
    ///     `"stop"` or `"length"` instead of `"tool_calls"` — OpenAI
    ///     spec violation).
    #[test]
    fn h200_sse_event_ordering_no_regression() {
        let src = include_str!("engine.rs");
        let fn_marker = "fn generate_stream_gemma4_once_slot_aware(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H200: generate_stream_gemma4_once_slot_aware not found");
        let fn_window = &src[fn_idx..(fn_idx + 60_000).min(src.len())];

        // (a) Terminal Done event present (the body must end every
        // successful decode path with a Done event).
        assert!(
            fn_window.contains("GenerationEvent::Done {"),
            "H200 FALSIFIED: slot-aware streaming fn body does NOT \
             emit a terminal `GenerationEvent::Done {{ .. }}` event \
             — SSE stream termination broken."
        );

        // (b) No NEW typed-error SSE Error path for the streaming
        // tool-call sub-deferral.  Pin the EXACT phrase that was
        // load-bearing for the iter-2-decode-C surviving sub-deferral
        // surface — its presence in a `send!(...Error(...))` call
        // would mean the typed-error abort was reinstated.
        assert!(
            !fn_window.contains("streaming tool-call body at SlotId(N>0) requested"),
            "H200 FALSIFIED: slot-aware streaming fn body still \
             contains the iter-2-decode-C surviving sub-deferral SSE \
             Error event phrase `streaming tool-call body at \
             SlotId(N>0) requested` — typed-error abort \
             reinstated."
        );

        // (c) finish_reason override present (H197 (d) transitivity).
        assert!(
            fn_window.contains("if saw_tool_call {")
                && fn_window.contains("finish_reason = \"tool_calls\""),
            "H200 FALSIFIED: slot-aware streaming fn body lacks the \
             `if saw_tool_call {{ finish_reason = \"tool_calls\"; }}` \
             override — OpenAI tool-calls finish_reason contract \
             broken."
        );
    }

    /// **H201 (skip-mode)** — Orthogonal surfaces UNCHANGED.  Embed-
    /// arm body still does NOT call forward_decode_slot_aware (H136 /
    /// H180 / H188 / H195 transitivity).  Qwen3VL UNCHANGED.  Sub-
    /// deferral label `iter-B4c-kernel-iter-2-decode-C-stream-tool-
    /// call per ADR-040 §6.1.39` is STILL grep-able in engine.rs as
    /// a doc-comment cite (H87 forward-pointer discoverability).
    /// ADR-040 §6.1.48 closure block exists in the ADR.
    #[test]
    fn h201_orthogonal_surfaces_unchanged_and_sub_deferral_doc_cite_preserved() {
        let src = include_str!("engine.rs");

        // (a) Embed-arm body still does NOT call forward_decode_slot_
        // aware (mirror of H180 / H188 / H195).
        let embed_marker = "fn embed_gemma4_slot_aware(";
        let embed_idx = src
            .find(embed_marker)
            .expect("H201: embed_gemma4_slot_aware not found");
        let embed_window = &src[embed_idx..(embed_idx + 10_000).min(src.len())];
        assert!(
            !embed_window.contains(".forward_decode_slot_aware("),
            "H201 FALSIFIED: Embed-arm fn body calls \
             `forward_decode_slot_aware`. The Embed-arm has NO \
             decode loop — calling forward_decode_slot_aware would \
             corrupt the L2-normalized embedding vector."
        );

        // (b) Surviving sub-deferral label STILL present in engine.rs
        // as doc-cite (H87 forward-pointer discoverability — required
        // even after the typed-error is removed, so operators can
        // grep for the historical scope-narrowing decision).
        let stream_tc_label =
            "iter-B4c-kernel-iter-2-decode-C-stream-tool-call per ADR-040 §6.1.39";
        assert!(
            src.contains(stream_tc_label),
            "H201 FALSIFIED: surviving sub-deferral label \
             `{stream_tc_label}` is NOT present in engine.rs. \
             iter-2-decode-C-stream-tool-call closure must preserve \
             the operator-grep'able forward pointer per H87 \
             discipline."
        );

        // (c) ADR-040 §6.1.48 closure block exists (the new closure
        // block landing this iter).
        let adr =
            crate::serve::api::engine::adr040_history_doc() /* iter-230 A1: §6.1.x moved to history (aeb6e87c) */;
        assert!(
            adr.contains("### 6.1.48"),
            "H201 FALSIFIED: ADR-040 §6.1.48 closure block not \
             found. iter-2-decode-C-stream-tool-call sub-deferral \
             cites point at a non-existent destination."
        );

        // (d) Qwen35 + Qwen3VL slot-aware surfaces UNCHANGED (mirror
        // of H136 / H199).
        for required in [
            "generate_qwen35_once_slot_aware",
            "generate_stream_qwen35_once_slot_aware",
        ] {
            assert!(
                src.contains(required),
                "H201 FALSIFIED: Qwen35 slot-aware surface \
                 `{required}` removed — iter-2-decode-C-stream-tool-\
                 call accidentally touched Qwen35 architecture."
            );
        }
    }
}

// ════════════════════════════════════════════════════════════════════════════
// ADR-040 Phase B iter-B4c-kernel iter-2-embed + iter-2-batched
// (Gemma 4 orthogonal forward paths slot-aware structural-N/A closures) —
// 2026-05-30
// ────────────────────────────────────────────────────────────────────────────
//
// Closes the two remaining iter-2-* sub-deferrals from §6.1.32's followups
// list (lines 2938-2939 of the ADR):
//
//   - **iter-B4c-kernel-iter-2-embed**: `forward_embed_last` slot-aware
//     port.  CLOSED as STRUCTURAL N/A — the Embed-arm SlotId(N>0) surface
//     is shipped via the orchestrator `embed_gemma4_slot_aware` (iter-4
//     §6.1.36), which calls `forward_prefill_with_soft_tokens_slot_aware`
//     (the iter-2A landing per §6.1.32 + iter-2B routing per §6.1.34) —
//     NOT `forward_embed_last`.  The worker-arm dispatch fork at
//     `engine.rs:5845` routes `slot_id != SlotId(0)` into
//     `embed_gemma4_slot_aware`; only SerialFifo + SlotId(0) reaches the
//     legacy `g.weights.forward_embed_last(&prompt_tokens, &mut g.ctx)`
//     dispatch at `engine.rs:6026`.  A hypothetical
//     `forward_embed_last_slot_aware` would be DEAD CODE — no caller.
//
//   - **iter-B4c-kernel-iter-2-batched**: `forward_prefill_batched`
//     slot-aware port.  CLOSED as STRUCTURAL N/A — the batched variant is
//     gated on `HF2Q_SERVE_BATCHED_PREFILL` and ONLY called from
//     `generate_once` + `generate_stream_once` (the SerialFifo + SlotId(0)
//     paths at `engine.rs:7802` + `:12676`).  All four slot-aware
//     orchestrators (`generate_gemma4_once_slot_aware`,
//     `generate_stream_gemma4_once_slot_aware`, `embed_gemma4_slot_aware`,
//     `generate_gemma4_once_with_soft_tokens_slot_aware`) call
//     `forward_prefill_with_soft_tokens_slot_aware` exclusively.  A
//     hypothetical `forward_prefill_batched_slot_aware` would be DEAD
//     CODE — no caller.
//
// Both closures are structurally-honest typed pins: the load-bearing
// `iter-B4c-kernel-iter-2-{embed,batched} per ADR-040 §6.1.49` substrings
// are preserved as doc-comment cites in `src/serve/forward_prefill.rs`
// (forward_embed_last) and `src/serve/forward_prefill_batched.rs`
// (forward_prefill_batched) so `grep "iter-2-embed per"` and `grep
// "iter-2-batched per"` discover the closure block.  The label substrings
// are INTENTIONALLY NOT inside `MultiSeqError::CapabilityUnsupported`
// constructors — the SlotId(N>0) routing is the orchestrator's
// responsibility at the call-graph layer above these fns, not these fns
// themselves.
//
// Tests (H202–H206):
//
//   H202 (skip-mode): iter-2-embed structural-N/A pin landed.  The
//                     `forward_embed_last` fn signature in
//                     forward_prefill.rs is UNCHANGED (no `slot_id` /
//                     `multi_seq_kv*` params).  The §6.1.49 forward-
//                     pointer doc cite is grep-able.
//   H203 (skip-mode): iter-2-batched structural-N/A pin landed.  The
//                     `forward_prefill_batched` fn signature in
//                     forward_prefill_batched.rs is UNCHANGED (no
//                     `slot_id` / `multi_seq_kv*` params).  The §6.1.49
//                     forward-pointer doc cite is grep-able.
//   H204 (skip-mode): per-slot byte isolation discipline preserved for
//                     the orthogonal surfaces.  The slot-aware
//                     orchestrators do NOT call `forward_embed_last` or
//                     `forward_prefill_batched` — they route through
//                     `forward_prefill_with_soft_tokens_slot_aware`
//                     exclusively (H188 / H195 transitivity).
//   H205 (skip-mode): SerialFifo byte-equivalence preserved.  Both
//                     `forward_embed_last` AND `forward_prefill_batched`
//                     STILL appear in their non-slot-aware engine.rs
//                     call sites (engine.rs:6026 + :7802 + :12676).
//   H206 (skip-mode): production-default surfaces UNCHANGED.  Qwen35 +
//                     Qwen3VL orthogonal-fn surfaces unchanged (no
//                     Qwen35-specific iter-2-embed / iter-2-batched
//                     mention).  ADR-040 §6.1.49 closure block exists.

#[cfg(test)]
mod adr040_phase_b_iter_b4c_kernel_iter2_embed_batched_gemma4_tests {
    // Skip-mode source-grep tests; intentionally NO `use super::*;`.

    /// **H202 (skip-mode)** — iter-2-embed structural-N/A pin landed.
    ///
    /// (a) `forward_embed_last` fn signature in forward_prefill.rs is
    ///     UNCHANGED: no `slot_id` / `multi_seq_kv*` params (mirror of
    ///     H86 sibling-signature-unchanged discipline applied to the
    ///     orthogonal embed surface).
    /// (b) Doc-comment cite `iter-B4c-kernel iter-2-embed structural-N/A
    ///     closure (2026-05-30, §6.1.49)` is grep-able in
    ///     forward_prefill.rs (H87 forward-pointer discoverability).
    /// (c) The label substring `iter-B4c-kernel-iter-2-embed per
    ///     ADR-040 §6.1.49` is INTENTIONALLY NOT inside a
    ///     `MultiSeqError::CapabilityUnsupported` constructor — the
    ///     iter-2-embed surface has no typed deferral (the SlotId(N>0)
    ///     Embed routing is the orchestrator `embed_gemma4_slot_aware`
    ///     responsibility, NOT this fn's).
    #[test]
    fn h202_iter_2_embed_structural_na_pin_landed_in_forward_prefill_rs() {
        let src = include_str!("../forward_prefill.rs");

        // (a) Sibling fn signature unchanged: locate `pub fn forward_embed_last(`
        // + extract the signature window up to `-> Result<Vec<f32>>` + assert
        // no slot_id / multi_seq_kv tokens appear.
        let fn_marker = "pub fn forward_embed_last(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H202: forward_embed_last signature not found");
        let sig_end = src[fn_idx..]
            .find("-> Result<Vec<f32>>")
            .map(|off| fn_idx + off + "-> Result<Vec<f32>>".len())
            .unwrap_or(fn_idx + 400);
        let sig_window = &src[fn_idx..sig_end.min(src.len())];
        assert!(
            !sig_window.contains("slot_id"),
            "H202 FALSIFIED: `forward_embed_last` signature contains \
             `slot_id`. iter-2-embed structural-N/A discipline broken \
             — this fn MUST remain non-slot-aware; SlotId(N>0) Embed \
             routing is the orchestrator `embed_gemma4_slot_aware`'s \
             responsibility per §6.1.36."
        );
        assert!(
            !sig_window.contains("multi_seq_kv"),
            "H202 FALSIFIED: `forward_embed_last` signature mentions \
             `multi_seq_kv`. iter-2-embed structural-N/A discipline \
             broken — this fn MUST NOT consume the multi-seq scaffold; \
             that's the orchestrator's job."
        );

        // (b) Forward-pointer doc cite grep-able (H87 discipline).
        let closure_cite =
            "iter-B4c-kernel iter-2-embed structural-N/A closure (2026-05-30, §6.1.49)";
        assert!(
            src.contains(closure_cite),
            "H202 FALSIFIED: forward_prefill.rs is missing the iter-2-embed \
             closure cite `{closure_cite}`.  The structural-N/A finding is \
             not discoverable to a future implementer who greps for \
             `iter-2-embed per` — H87 discipline violated."
        );
        let short_label = "iter-B4c-kernel-iter-2-embed per ADR-040 §6.1.49";
        assert!(
            src.contains(short_label),
            "H202 FALSIFIED: forward_prefill.rs is missing the short \
             forward-pointer label `{short_label}`.  `grep \"iter-2-embed \
             per\"` would not discover the closure block."
        );

        // (c) The label substring is NOT inside a typed CapabilityUnsupported
        // constructor — pin the absence of the negative pattern.  iter-2-embed
        // has no typed deferral; the only `iter-2-embed per` appearances are
        // doc-comment cites.
        assert!(
            !src.contains(
                "CapabilityUnsupported { capability: \"gemma4-forward-embed-last-slot-N (iter-B4c-kernel-iter-2-embed"
            ),
            "H202 FALSIFIED: forward_prefill.rs contains a typed \
             `MultiSeqError::CapabilityUnsupported` for the iter-2-embed \
             surface.  iter-2-embed is structural-N/A — there is no \
             typed deferral to surface; the SlotId(N>0) Embed routing is \
             `embed_gemma4_slot_aware` (§6.1.36) at the orchestrator \
             layer above this fn."
        );
    }

    /// **H203 (skip-mode)** — iter-2-batched structural-N/A pin landed.
    ///
    /// (a) `forward_prefill_batched` fn signature in
    ///     forward_prefill_batched.rs is UNCHANGED: no `slot_id` /
    ///     `multi_seq_kv*` params.
    /// (b) Doc-comment cite `iter-B4c-kernel iter-2-batched
    ///     structural-N/A closure (2026-05-30, §6.1.49)` is grep-able
    ///     (H87 forward-pointer discoverability).
    /// (c) The label substring `iter-B4c-kernel-iter-2-batched per
    ///     ADR-040 §6.1.49` is INTENTIONALLY NOT inside a
    ///     `MultiSeqError::CapabilityUnsupported` constructor.
    #[test]
    fn h203_iter_2_batched_structural_na_pin_landed_in_forward_prefill_batched_rs() {
        let src = include_str!("../forward_prefill_batched.rs");

        // (a) Sibling fn signature unchanged.
        let fn_marker = "pub fn forward_prefill_batched(";
        let fn_idx = src
            .find(fn_marker)
            .expect("H203: forward_prefill_batched signature not found");
        let sig_end = src[fn_idx..]
            .find("-> Result<u32>")
            .map(|off| fn_idx + off + "-> Result<u32>".len())
            .unwrap_or(fn_idx + 600);
        let sig_window = &src[fn_idx..sig_end.min(src.len())];
        assert!(
            !sig_window.contains("slot_id"),
            "H203 FALSIFIED: `forward_prefill_batched` signature contains \
             `slot_id`. iter-2-batched structural-N/A discipline broken — \
             this fn MUST remain non-slot-aware; the batched variant is \
             orthogonal to per-request slot routing (HF2Q_SERVE_BATCHED_\
             PREFILL gate; SerialFifo + SlotId(0) only)."
        );
        assert!(
            !sig_window.contains("multi_seq_kv"),
            "H203 FALSIFIED: `forward_prefill_batched` signature mentions \
             `multi_seq_kv`. iter-2-batched structural-N/A discipline \
             broken — this fn MUST NOT consume the multi-seq scaffold."
        );

        // (b) Forward-pointer doc cite grep-able (H87 discipline).
        let closure_cite =
            "iter-B4c-kernel iter-2-batched structural-N/A closure (2026-05-30, §6.1.49)";
        assert!(
            src.contains(closure_cite),
            "H203 FALSIFIED: forward_prefill_batched.rs is missing the \
             iter-2-batched closure cite `{closure_cite}`.  H87 discipline \
             violated."
        );
        let short_label = "iter-B4c-kernel-iter-2-batched per ADR-040 §6.1.49";
        assert!(
            src.contains(short_label),
            "H203 FALSIFIED: forward_prefill_batched.rs is missing the \
             short forward-pointer label `{short_label}`.  `grep \"iter-\
             2-batched per\"` would not discover the closure block."
        );

        // (c) No typed CapabilityUnsupported for the iter-2-batched surface.
        assert!(
            !src.contains(
                "CapabilityUnsupported { capability: \"gemma4-forward-prefill-batched-slot-N (iter-B4c-kernel-iter-2-batched"
            ),
            "H203 FALSIFIED: forward_prefill_batched.rs contains a typed \
             `MultiSeqError::CapabilityUnsupported` for the iter-2-batched \
             surface.  iter-2-batched is structural-N/A — the slot-aware \
             orchestrators bypass this fn entirely (they call \
             `forward_prefill_with_soft_tokens_slot_aware`)."
        );
    }

    /// **H204 (skip-mode)** — per-slot byte isolation discipline
    /// preserved for the orthogonal surfaces.
    ///
    /// (a) None of the 4 Gemma 4 slot-aware orchestrators
    ///     (`generate_gemma4_once_slot_aware`,
    ///     `generate_stream_gemma4_once_slot_aware`,
    ///     `embed_gemma4_slot_aware`,
    ///     `generate_gemma4_once_with_soft_tokens_slot_aware`) call
    ///     `forward_embed_last` (mirror of H188 / H201 transitivity).
    /// (b) None of the 4 slot-aware orchestrators call
    ///     `forward_prefill_batched`.
    ///
    /// Note: this test must scan ONLY executable code lines (not doc
    /// comments) — the slot-aware orchestrators carry copious doc-cites
    /// mentioning `g.weights.forward_embed_last(&prompt_tokens, &mut
    /// g.ctx)` as the legacy SerialFifo dispatch they SHORT-CIRCUIT
    /// (see embed_gemma4_slot_aware docstring at engine.rs:9838).  The
    /// test strips `///` doc-comment lines + `//` regular-comment lines
    /// before checking for the bypass pattern, so the H87 forward-
    /// pointer discoverability discipline is preserved.
    #[test]
    fn h204_slot_aware_orchestrators_bypass_orthogonal_fns() {
        let src = include_str!("engine.rs");

        for orchestrator_marker in [
            "fn generate_gemma4_once_slot_aware(",
            "fn generate_stream_gemma4_once_slot_aware(",
            "fn embed_gemma4_slot_aware(",
            "fn generate_gemma4_once_with_soft_tokens_slot_aware(",
        ] {
            let orch_idx = src.find(orchestrator_marker).unwrap_or_else(|| {
                panic!("H204: slot-aware orchestrator `{orchestrator_marker}` not found");
            });
            // Find the orchestrator body's end by brace-matching from
            // the opening `{` after the marker.  Falls back to a 75 KB
            // window if the brace-match fails (defense-in-depth — should
            // never engage at runtime per the well-formed source tree).
            let body_start = src[orch_idx..]
                .find('{')
                .map(|off| orch_idx + off + 1)
                .unwrap_or(orch_idx);
            let body_end = {
                let bytes = src.as_bytes();
                let mut depth: i32 = 1;
                let mut i = body_start;
                while i < bytes.len() && depth > 0 {
                    match bytes[i] {
                        b'{' => depth += 1,
                        b'}' => depth -= 1,
                        _ => {}
                    }
                    i += 1;
                }
                if depth == 0 {
                    i
                } else {
                    (orch_idx + 75_000).min(src.len())
                }
            };
            let orch_window = &src[orch_idx..body_end.min(src.len())];

            // Strip doc-comment + regular-comment lines so the source-grep
            // checks only executable code.  H87 forward-pointer
            // discoverability discipline is preserved (the cites in
            // docstrings still appear in the raw source via the H202 / H203
            // grep paths).
            let code_only: String = orch_window
                .lines()
                .filter(|line| {
                    let trimmed = line.trim_start();
                    !trimmed.starts_with("///")
                        && !trimmed.starts_with("//!")
                        && !trimmed.starts_with("//")
                })
                .collect::<Vec<_>>()
                .join("\n");

            // (a) MUST NOT call forward_embed_last on `.weights.` (which
            // would mean it's bypassing embed_gemma4_slot_aware's
            // forward_prefill_with_soft_tokens_slot_aware routing).
            assert!(
                !code_only.contains(".weights.forward_embed_last("),
                "H204 FALSIFIED: orchestrator `{orchestrator_marker}` calls \
                 `.weights.forward_embed_last(...)` — bypassing the \
                 slot-aware routing through \
                 `forward_prefill_with_soft_tokens_slot_aware`.  \
                 iter-2-embed structural-N/A discipline broken; the \
                 SlotId(N>0) embed path MUST route through \
                 `embed_gemma4_slot_aware` per §6.1.36 (which calls \
                 `forward_prefill_with_soft_tokens_slot_aware`, NOT \
                 `forward_embed_last`)."
            );

            // (b) MUST NOT call forward_prefill_batched on `.weights.`.
            assert!(
                !code_only.contains(".forward_prefill_batched("),
                "H204 FALSIFIED: orchestrator `{orchestrator_marker}` \
                 calls `.forward_prefill_batched(...)` — bypassing the \
                 slot-aware routing through \
                 `forward_prefill_with_soft_tokens_slot_aware`.  \
                 iter-2-batched structural-N/A discipline broken; the \
                 batched variant is orthogonal to per-request slot \
                 routing (SerialFifo / SlotId(0) only)."
            );
        }
    }

    /// **H205 (skip-mode)** — SerialFifo byte-equivalence preserved.
    ///
    /// Both `forward_embed_last` AND `forward_prefill_batched` STILL
    /// appear in their non-slot-aware engine.rs call sites — the
    /// SerialFifo / SlotId(0) production paths are untouched.
    ///
    /// (a) `g.weights.forward_embed_last(&prompt_tokens, &mut g.ctx)`
    ///     call site at engine.rs:6026 STILL present (the legacy
    ///     SerialFifo + SlotId(0) Embed dispatch — the `slot_id !=
    ///     SlotId(0)` predicate at engine.rs:5845 short-circuits this
    ///     for SlotAware + SlotId(N>0) only).
    /// (b) `.forward_prefill_batched(prompt_tokens, max_tokens, 0, &mut
    ///     loaded.ctx)` call site at engine.rs:7802 + :12676 STILL
    ///     present (the SerialFifo + SlotId(0) generate_once +
    ///     generate_stream_once dispatch — the slot-aware orchestrators
    ///     bypass these entirely).
    #[test]
    fn h205_serial_fifo_call_sites_preserved() {
        let src = include_str!("engine.rs");

        // (a) Legacy Embed dispatch preserved (the SerialFifo + SlotId(0)
        // production code path).
        assert!(
            src.contains("g.weights.forward_embed_last(&prompt_tokens, &mut g.ctx)"),
            "H205 FALSIFIED: engine.rs no longer contains the legacy \
             SerialFifo + SlotId(0) Embed dispatch \
             `g.weights.forward_embed_last(&prompt_tokens, &mut g.ctx)`. \
             iter-2-embed accidentally regressed the non-slot-aware path."
        );

        // (b) Batched prefill dispatch preserved (both generate_once and
        // generate_stream_once should still contain a forward_prefill_batched
        // call).  At least 2 call sites are required (generate_once
        // engine.rs:7802 + generate_stream_once engine.rs:12676).
        let batched_call_count = src.matches(".forward_prefill_batched(").count();
        assert!(
            batched_call_count >= 2,
            "H205 FALSIFIED: engine.rs has only {batched_call_count} \
             call site(s) of `.forward_prefill_batched(` — expected ≥2 \
             (generate_once + generate_stream_once).  iter-2-batched \
             accidentally regressed the SerialFifo + SlotId(0) batched \
             prefill dispatch."
        );
    }

    /// **H206 (skip-mode)** — production-default surfaces UNCHANGED +
    /// §6.1.49 closure block exists.
    ///
    /// (a) Qwen35 + Qwen3VL orthogonal-fn surfaces unchanged — the Qwen35
    ///     architecture source files do NOT mention the Gemma-specific
    ///     iter-2-embed / iter-2-batched labels (these are Gemma 4 only
    ///     sub-deferrals on the iter-B4c-kernel arc).
    /// (b) Qwen35 slot-aware fn surface UNCHANGED (mirror of H199 / H201
    ///     transitivity).
    /// (c) ADR-040 §6.1.49 closure block exists (the new closure block
    ///     landing this iter — forward-pointer destination required for
    ///     the H202 + H203 short-label cites).
    /// (d) Surviving sub-deferral labels `iter-2-embed` + `iter-2-batched`
    ///     in the ADR are MARKED SHIPPED (the §6.1.32 followups list +
    ///     all subsequent closure blocks that historically said
    ///     "(UNCHANGED from §6.1.32)" should now point at §6.1.49 for
    ///     the SHIPPED status).
    #[test]
    fn h206_production_default_surfaces_unchanged_and_adr_closure_landed() {
        let engine_src = include_str!("engine.rs");

        // (a) Qwen35 architecture sources do NOT mention iter-2-embed /
        // iter-2-batched (these are Gemma 4 only labels on the
        // iter-B4c-kernel arc — iter-C2d-cont-kernel is the Qwen35 arc).
        let qwen35_forward_gpu = include_str!("../../inference/models/qwen35/forward_gpu.rs");
        assert!(
            !qwen35_forward_gpu.contains("iter-2-embed per ADR-040"),
            "H206 FALSIFIED: Qwen35 forward_gpu.rs mentions \
             `iter-2-embed per ADR-040`. The iter-2-embed scope is \
             Gemma 4 only — Qwen35 architecture accidentally touched."
        );
        assert!(
            !qwen35_forward_gpu.contains("iter-2-batched per ADR-040"),
            "H206 FALSIFIED: Qwen35 forward_gpu.rs mentions \
             `iter-2-batched per ADR-040`. The iter-2-batched scope is \
             Gemma 4 only — Qwen35 architecture accidentally touched."
        );

        // (b) Qwen35 slot-aware fn surface STILL defined (mirror of
        // H199 / H201).
        for required in [
            "generate_qwen35_once_slot_aware(",
            "embed_qwen35_slot_aware(",
        ] {
            assert!(
                engine_src.contains(required),
                "H206 FALSIFIED: Qwen35 slot-aware fn `{required}` is \
                 NOT present — iter-2-embed + iter-2-batched \
                 accidentally regressed a Qwen35 lift."
            );
        }

        // (c) ADR-040 §6.1.49 closure block exists.
        let adr =
            crate::serve::api::engine::adr040_history_doc() /* iter-230 A1: §6.1.x moved to history (aeb6e87c) */;
        assert!(
            adr.contains("### 6.1.49"),
            "H206 FALSIFIED: ADR-040 §6.1.49 closure block not found. \
             iter-2-embed + iter-2-batched sub-deferral cites point at \
             a non-existent destination; H202 + H203 short-label cites \
             would dangle."
        );

        // (d) iter-2-embed + iter-2-batched substrings still grep-able
        // in ADR (forward-pointer discoverability — must remain even
        // after SHIPPED).
        assert!(
            adr.contains("iter-2-embed"),
            "H206 FALSIFIED: ADR-040 no longer mentions `iter-2-embed`. \
             Historical scope-narrowing decision lost."
        );
        assert!(
            adr.contains("iter-2-batched"),
            "H206 FALSIFIED: ADR-040 no longer mentions \
             `iter-2-batched`. Historical scope-narrowing decision lost."
        );

        // (e) §6.1.49 closure block names BOTH iter-2-embed AND
        // iter-2-batched as SHIPPED structural-N/A — pin both
        // substrings INSIDE the §6.1.49 block to defend against a
        // partial-rename regression.
        let section_idx = adr
            .find("### 6.1.49")
            .expect("H206 (e): ADR-040 §6.1.49 closure block missing");
        // Section bounded by the next `### 6.1.` or end-of-file.
        let section_end_rel = adr[section_idx + 10..]
            .find("\n### ")
            .unwrap_or(adr.len() - section_idx - 10);
        let section_window = &adr[section_idx..(section_idx + 10 + section_end_rel).min(adr.len())];
        assert!(
            section_window.contains("iter-2-embed"),
            "H206 FALSIFIED: §6.1.49 closure block does NOT name \
             `iter-2-embed` — the closure is incomplete."
        );
        assert!(
            section_window.contains("iter-2-batched"),
            "H206 FALSIFIED: §6.1.49 closure block does NOT name \
             `iter-2-batched` — the closure is incomplete."
        );
        assert!(
            section_window.contains("structural"),
            "H206 FALSIFIED: §6.1.49 closure block does NOT contain \
             the word `structural` — the structural-N/A finding is \
             not declared."
        );
    }
}

// ────────────────────────────────────────────────────────────────────
// ADR-040 §6.1.50 — iter-C2d-cont-kernel-iter-LCP + iter-G (Qwen35)
//                  + iter-B4c-kernel-iter-2D-lcp (Gemma 4) joint closure
//                  (2026-05-30)
// ────────────────────────────────────────────────────────────────────
//
// Closes the 3 remaining orthogonal orchestrator-side perf optimization
// deferrals on the iter-C2d-cont-kernel + iter-B4c-kernel arcs.
//
//   * iter-C2d-cont-kernel-iter-LCP (Qwen35 slot-aware LCP / chunked-
//     prefill snapshot codec): STRUCTURAL N/A.  The snapshot codec keys
//     snapshots on per-request `max_seq_len = prompt_len + max_tokens +
//     64` while the persistent multi-seq cache is sized to
//     `cfg.max_position_embeddings`.  Cross-slot prefix sharing carries
//     tenant-isolation risk (LCP cache is global, slot regions are per-
//     tenant).  Full-equality prompt-cache HITs already use
//     `restore_partial(snap, prompt_len)` (working) — this IS the LCP
//     fast-path operators get in slot-aware mode.  The remaining
//     chunked-prefill mid-store + cross-request `probe_lcp_opportunity`
//     paths would require multi-iter snapshot-codec extensions beyond
//     the iter-LCP scope.
//
//   * iter-C2d-cont-kernel-iter-G (Qwen35 forward_gpu_greedy slot-aware
//     fast-path): REAL LIFT.  `forward_gpu_greedy` accepts `slot_id`
//     since B4d §6.1.44 (2026-05-30).  iter-G ports the 4 slot-aware fn
//     greedy-only decode branches from `forward_gpu_last_logits +
//     greedy_argmax_last_token` to `forward_gpu_greedy(.., slot_id)` —
//     saves ~250 µs per step at vocab=151k by skipping the F32 readback.
//     Sampling + logprobs branches UNCHANGED.
//
//   * iter-B4c-kernel-iter-2D-lcp (Gemma 4 dense F32 LCP partial-prefix
//     slot-aware port): STRUCTURAL N/A.  The LCP path consumes cached
//     `Arc<DenseKvBuffers>` into `self.dense_kvs` (`engine.rs:7593`);
//     the iter-2D slot-aware path mounts slot-views into the SAME
//     `self.dense_kvs` field — MUTUALLY EXCLUSIVE mount sources.  Plus
//     the same global-vs-per-tenant isolation concern as Qwen35
//     iter-LCP.
//
// Tests (H207–H212):
//
//   H207 (skip-mode): iter-LCP STRUCTURAL N/A — both `generate_qwen35_
//                     once_slot_aware` AND `generate_stream_qwen35_
//                     once_extended_slot_aware` AND `embed_qwen35_
//                     slot_aware` AND `generate_qwen35_once_with_soft_
//                     tokens_slot_aware` docstrings contain the
//                     iter-C2d-cont-kernel-iter-LCP STRUCTURAL N/A pin
//                     + the §6.1.50 forward-pointer cite.  Label
//                     substring NOT inside `MultiSeqError::Capability
//                     Unsupported` constructor.
//   H208 (skip-mode): iter-G REAL LIFT — 4 Qwen35 slot-aware fns each
//                     call `forward_gpu_greedy` with `slot_id` in the
//                     greedy decode branch.  Source-grep witness.
//   H209 (skip-mode): iter-2D-lcp STRUCTURAL N/A — forward_prefill.rs
//                     iter-2D dense branch contains the §6.1.50 forward-
//                     pointer cite + STRUCTURAL N/A pin.  Label
//                     substring NOT inside `MultiSeqError::Capability
//                     Unsupported` constructor.
//   H210 (skip-mode): SerialFifo + SlotId(0) byte-equivalence preserved
//                     — non-slot-aware `generate_qwen35_once` still uses
//                     `forward_gpu_greedy(.., SlotId(0))` at decode
//                     (already does); the slot-aware sites' iter-G
//                     lifts are at `slot_id` (NOT hard-coded SlotId(0)).
//   H211 (skip-mode): Qwen35 + Qwen3VL surfaces UNCHANGED — `forward_
//                     gpu_greedy` signature still accepts `slot_id`
//                     (B4d §6.1.44 preserved); no Gemma 4 slot-aware
//                     fns gain iter-G ports (Gemma 4 uses `forward_
//                     decode_slot_aware` which is internally greedy).
//   H212 (skip-mode): production-default sampling paths UNCHANGED —
//                     the sampling + logprobs branches in slot-aware
//                     fns still use `forward_gpu_last_logits +
//                     sample_logits_qwen35[_with_logprob]`.

#[cfg(test)]
mod adr040_phase_c_iter_c2d_cont_kernel_iter_lcp_g_qwen35_iter_2d_lcp_gemma4_tests {
    // Skip-mode source-grep tests; intentionally NO `use super::*;`.

    /// **H207 (skip-mode)** — iter-C2d-cont-kernel-iter-LCP STRUCTURAL
    /// N/A pin landed in all 4 Qwen35 slot-aware fns (Generate +
    /// GenerateStream + Embed + SoftTokens — the deepstack soft-tokens
    /// fn shares the SoftTokens docstring narrative).
    ///
    /// (a) Each fn's docstring contains the §6.1.50 forward-pointer
    ///     cite `iter-C2d-cont-kernel-iter-LCP per ADR-040 §6.1.50` +
    ///     the phrase `STRUCTURAL N/A`.
    /// (b) The label substring is INTENTIONALLY NOT inside a
    ///     `MultiSeqError::CapabilityUnsupported` constructor — mirror
    ///     of §6.1.49 iter-2-embed / iter-2-batched discipline.
    #[test]
    fn h207_iter_lcp_structural_na_pin_landed_in_qwen35_slot_aware_fns() {
        let src = include_str!("engine_qwen35.rs");

        // (a) Forward-pointer cite at the docstring level.
        let lcp_cite = "iter-C2d-cont-kernel-iter-LCP per ADR-040 §6.1.50";
        let cite_count = src.matches(lcp_cite).count();
        assert!(
            cite_count >= 4,
            "H207 FALSIFIED: iter-C2d-cont-kernel-iter-LCP per ADR-040 \
             §6.1.50 cite count {cite_count} < 4 (one per slot-aware \
             fn: Generate + GenerateStream + Embed + SoftTokens). The \
             STRUCTURAL N/A closure block must add the forward-pointer \
             cite at each slot-aware fn's docstring for H87 discover- \
             ability."
        );

        // (b) Phrase `STRUCTURAL N/A` present at least 4× (one per fn).
        let phrase_count = src.matches("STRUCTURAL N/A").count();
        assert!(
            phrase_count >= 4,
            "H207 FALSIFIED: `STRUCTURAL N/A` phrase count \
             {phrase_count} < 4.  The structural-N/A finding must be \
             declared verbatim in each slot-aware fn's docstring."
        );

        // (c) Label substring NOT inside a typed `CapabilityUnsupported`
        // constructor — mirror of §6.1.49 H202 discipline.  Scan for
        // any occurrence of `CapabilityUnsupported` within 200 chars
        // BEFORE the cite — none allowed.
        for (i, _) in src.match_indices(lcp_cite) {
            let window_start = i.saturating_sub(400);
            let window = &src[window_start..i];
            assert!(
                !window.contains("CapabilityUnsupported {"),
                "H207 FALSIFIED: the iter-LCP cite at offset {i} is \
                 inside a `CapabilityUnsupported {{` constructor \
                 within 400 chars — STRUCTURAL N/A discipline broken \
                 (typed deferrals NOT allowed for STRUCTURAL N/A \
                 closures; mirror of §6.1.49 H202 forbidden pattern)."
            );
        }
    }

    /// **H208 (skip-mode)** — iter-C2d-cont-kernel-iter-G REAL LIFT
    /// landed at all 4 Qwen35 slot-aware fn greedy decode branches.
    ///
    /// (a) `forward_gpu_greedy` is called from `engine_qwen35.rs` at
    ///     ≥4 NEW sites (one per slot-aware fn).
    /// (b) Each iter-G call site passes `slot_id` (NOT hard-coded
    ///     SlotId(0)).
    /// (c) The §6.1.50 iter-G cite appears at ≥4 sites for H87
    ///     discoverability.
    #[test]
    fn h208_iter_g_real_lift_landed_in_qwen35_slot_aware_fns() {
        let src = include_str!("engine_qwen35.rs");

        // (a) + (b) iter-G witness: scan for ".forward_gpu_greedy("
        // call sites AND walk forward up to 600 chars to find the
        // `slot_id` arg.  Pre-iter-G there is exactly 1 existing
        // forward_gpu_greedy call site (in `generate_qwen35_once` at
        // engine_qwen35.rs:2077 — the SerialFifo + SlotId(0) path).
        // Post-iter-G we expect ≥5 sites total (1 pre-existing + 4 iter-G
        // landings: generate_qwen35_once_slot_aware decode +
        // generate_stream_qwen35_once_extended_slot_aware decode +
        // generate_qwen35_once_with_soft_tokens_slot_aware decode +
        // generate_qwen35_once_with_soft_tokens_and_deepstack_slot_aware
        // decode).
        let greedy_marker = ".forward_gpu_greedy(";
        let greedy_count = src.matches(greedy_marker).count();
        assert!(
            greedy_count >= 5,
            "H208 FALSIFIED: `.forward_gpu_greedy(` call-site count \
             {greedy_count} < 5 (1 pre-existing in `generate_qwen35_once` \
             at engine_qwen35.rs:~2077 + 4 NEW iter-G landings — one per \
             slot-aware fn).  iter-G REAL LIFT did not land — verify the \
             4 slot-aware fns' greedy decode branches actually route \
             through `forward_gpu_greedy(.., slot_id)`."
        );

        // (c) Each NEW iter-G site cites `ADR-040 §6.1.50 iter-G` for
        // H87 discoverability.
        let iter_g_cite_count = src.matches("ADR-040 §6.1.50 iter-G").count();
        assert!(
            iter_g_cite_count >= 4,
            "H208 FALSIFIED: `ADR-040 §6.1.50 iter-G` cite count \
             {iter_g_cite_count} < 4 — at least one slot-aware fn's \
             iter-G call site is missing the forward-pointer cite."
        );

        // (d) The greedy fast-path slot-aware port comment from
        // §6.1.27/§6.1.28/§6.1.30 ("greedy fast-path slot-aware port is
        // iter-C2d-cont-kernel-iter-G") should be GONE from inside the
        // slot-aware fn bodies — replaced with the real lift.  We can't
        // search for absolute absence (the section closure block adds
        // its own narrative), but we can pin that the pre-iter-G
        // comment marker "but iter-1 keeps the simpler forward_gpu_last_
        // logits dispatch for minimal LOC delta" is REMOVED.  That
        // exact phrasing was the pre-§6.1.50 marker.
        assert!(
            !src.contains(
                "keeps the simpler forward_gpu_last_logits dispatch for minimal LOC delta"
            ),
            "H208 FALSIFIED: the pre-iter-G marker comment \
             `keeps the simpler forward_gpu_last_logits dispatch for \
             minimal LOC delta` is STILL present in engine_qwen35.rs. \
             The iter-1 docstring narration of the deferred-greedy- \
             fast-path was NOT cleaned up when iter-G landed — drift \
             between the docstring and the body."
        );
    }

    /// **H209 (skip-mode)** — iter-B4c-kernel-iter-2D-lcp STRUCTURAL
    /// N/A pin landed in `forward_prefill.rs` at the iter-2D dense F32
    /// branch.
    ///
    /// (a) The §6.1.50 forward-pointer cite
    ///     `iter-B4c-kernel-iter-2D-lcp per ADR-040 §6.1.50` is grep-
    ///     able in `forward_prefill.rs`.
    /// (b) The phrase `STRUCTURAL N/A` appears in the iter-2D branch
    ///     body.
    /// (c) The label substring is INTENTIONALLY NOT inside a
    ///     `MultiSeqError::CapabilityUnsupported` constructor (mirror
    ///     of §6.1.49 iter-2-embed discipline).
    #[test]
    fn h209_iter_2d_lcp_structural_na_pin_landed_in_forward_prefill_rs() {
        let src = include_str!("../forward_prefill.rs");

        // (a) Forward-pointer cite.
        let lcp_cite = "iter-B4c-kernel-iter-2D-lcp per ADR-040 §6.1.50";
        assert!(
            src.contains(lcp_cite),
            "H209 FALSIFIED: the §6.1.50 iter-2D-lcp forward-pointer \
             cite `{lcp_cite}` is NOT present in `forward_prefill.rs`. \
             The STRUCTURAL N/A closure must preserve the operator- \
             grep'able forward pointer per H87 discipline."
        );

        // (b) `STRUCTURAL N/A` declared at the iter-2D branch site.
        // The Qwen35 STRUCTURAL N/A phrase lives in `engine_qwen35.rs`;
        // this assertion is for the Gemma 4 iter-2D-lcp landing in
        // `forward_prefill.rs`.
        let cite_idx = src
            .find(lcp_cite)
            .expect("H209 (b): cite was just asserted present above");
        let window_end = (cite_idx + 1500).min(src.len());
        let window = &src[cite_idx.saturating_sub(1500)..window_end];
        assert!(
            window.contains("STRUCTURAL N/A"),
            "H209 FALSIFIED: the iter-2D-lcp STRUCTURAL N/A pin near \
             the §6.1.50 cite is missing the phrase `STRUCTURAL N/A`. \
             The structural-N/A finding must be declared verbatim."
        );

        // (c) Label substring NOT inside a `CapabilityUnsupported`
        // constructor — scan within 400 chars before the cite.
        let window_start = cite_idx.saturating_sub(400);
        let pre_window = &src[window_start..cite_idx];
        assert!(
            !pre_window.contains("CapabilityUnsupported {"),
            "H209 FALSIFIED: the iter-2D-lcp cite is inside a \
             `CapabilityUnsupported {{` constructor — STRUCTURAL N/A \
             discipline broken (typed deferrals NOT allowed for \
             STRUCTURAL N/A closures; mirror of §6.1.49 H203 forbidden \
             pattern)."
        );
    }

    /// **H210 (skip-mode)** — SerialFifo + SlotId(0) byte-equivalence
    /// preserved for the iter-G real lift.
    ///
    /// (a) The pre-existing `forward_gpu_greedy(.., SlotId(0))` call
    ///     site in `generate_qwen35_once` (engine_qwen35.rs:~2077) is
    ///     STILL present — SerialFifo + SlotId(0) decode path unchanged.
    /// (b) The iter-G lift call sites in the 4 slot-aware fns pass
    ///     `slot_id` (NOT hard-coded SlotId(0)).
    #[test]
    fn h210_serial_fifo_byte_equivalence_preserved_for_iter_g() {
        let src = include_str!("engine_qwen35.rs");

        // (a) SerialFifo pre-existing decode call site unchanged.
        let serial_marker =
            ".forward_gpu_greedy(&[next_token], &decode_positions, &mut kv_cache, SlotId(0))";
        assert!(
            src.contains(serial_marker),
            "H210 FALSIFIED: the pre-iter-G `forward_gpu_greedy(.., \
             &mut kv_cache, SlotId(0))` call in `generate_qwen35_once` \
             at engine_qwen35.rs:~2077 is REMOVED.  SerialFifo + \
             SlotId(0) decode byte-equivalence (H51 / H1 / H2 chain) \
             BROKEN — the non-slot-aware path must NOT be touched by \
             iter-G."
        );

        // (b) iter-G call sites pass `slot_id` — count call sites that
        // pass an arg named `slot_id` immediately.  We grep for the
        // pattern `forward_gpu_greedy(` + walk ahead to find `slot_id`
        // BEFORE the closing `)` of the call.  Number of such sites
        // should be ≥4.
        let mut iter_g_slot_id_sites = 0usize;
        let mut search_from = 0usize;
        while let Some(off) = src[search_from..].find(".forward_gpu_greedy(") {
            let abs = search_from + off;
            // Walk forward depth-balancing parens to find the matching
            // close paren of THIS call.
            let mut depth = 0i32;
            let mut close_off: Option<usize> = None;
            for (i, ch) in src[abs..].char_indices() {
                match ch {
                    '(' => depth += 1,
                    ')' => {
                        depth -= 1;
                        if depth == 0 {
                            close_off = Some(abs + i + 1);
                            break;
                        }
                    }
                    _ => {}
                }
                if i > 2000 {
                    break;
                }
            }
            let after = close_off.unwrap_or((abs + 600).min(src.len()));
            let call_window = &src[abs..after];
            // The pre-existing SerialFifo call is on one line and
            // contains the literal `SlotId(0)`; iter-G call sites pass
            // the local var `slot_id`.
            let is_serial_fifo_site = call_window.contains("SlotId(0)");
            let mentions_slot_id_var = call_window.contains("slot_id,")
                || call_window.contains("slot_id\n")
                || call_window.contains("slot_id)");
            if !is_serial_fifo_site && mentions_slot_id_var {
                iter_g_slot_id_sites += 1;
            }
            search_from = abs + 1;
        }
        assert!(
            iter_g_slot_id_sites >= 4,
            "H210 FALSIFIED: iter-G call sites passing `slot_id` \
             (NOT `SlotId(0)`) count {iter_g_slot_id_sites} < 4 — \
             at least one slot-aware fn's iter-G call site is \
             accidentally hard-coding `SlotId(0)`, breaking the \
             per-slot routing contract."
        );
    }

    /// **H211 (skip-mode)** — Qwen35 + Qwen3VL surfaces UNCHANGED by
    /// iter-LCP / iter-G / iter-2D-lcp.
    ///
    /// (a) `forward_gpu_greedy` signature still accepts `slot_id:
    ///     SlotId` (B4d §6.1.44 contract preserved — H167 transitivity).
    /// (b) No Gemma 4 slot-aware fn gains an `iter-G` real lift — Gemma
    ///     4 uses `forward_decode_slot_aware` which is internally
    ///     greedy at the kernel level (no separate
    ///     `forward_gpu_greedy` analog).
    /// (c) The `iter-B4c-kernel-iter-G` label is still listed in the
    ///     remaining followups (it's an orchestrator-side perf
    ///     optimization mirror of Qwen35 iter-G, NOT yet landed for
    ///     Gemma 4 in this iter).
    #[test]
    fn h211_qwen35_qwen3vl_surfaces_unchanged_by_iter_lcp_g_and_iter_2d_lcp() {
        let src_qwen35_forward =
            include_str!("../../../src/inference/models/qwen35/forward_gpu.rs");

        // (a) `forward_gpu_greedy` signature still accepts `slot_id:
        // SlotId` — B4d §6.1.44 H167 transitivity.
        let fn_marker = "pub fn forward_gpu_greedy(";
        let fn_idx = src_qwen35_forward
            .find(fn_marker)
            .expect("H211 (a): `forward_gpu_greedy` declaration not found");
        let sig_end = src_qwen35_forward[fn_idx..]
            .find(") -> Result<u32>")
            .map(|off| fn_idx + off + ") -> Result<u32>".len())
            .unwrap_or(fn_idx + 1000);
        let sig_window = &src_qwen35_forward[fn_idx..sig_end.min(src_qwen35_forward.len())];
        assert!(
            sig_window.contains("slot_id: SlotId"),
            "H211 FALSIFIED: `forward_gpu_greedy` signature in qwen35/\
             forward_gpu.rs NO LONGER accepts `slot_id: SlotId`.  B4d \
             §6.1.44 H167 contract BROKEN — iter-G regressed the \
             upstream signature."
        );

        // (b) No Gemma 4 slot-aware fn calls `forward_gpu_greedy(`
        // (that fn is Qwen35-specific; Gemma 4 uses
        // `forward_decode_slot_aware` which is internally greedy).
        let src_engine = include_str!("engine.rs");
        let g4_marker = ".forward_gpu_greedy(";
        let g4_slot_aware_section = src_engine
            .find("fn generate_gemma4_once_slot_aware(")
            .map(|idx| &src_engine[idx..(idx + 200_000).min(src_engine.len())])
            .unwrap_or("");
        assert!(
            !g4_slot_aware_section.contains(g4_marker),
            "H211 FALSIFIED: a Gemma 4 slot-aware fn calls \
             `forward_gpu_greedy(` — that fn is Qwen35-specific.  \
             Gemma 4's iter-G is a separate orchestrator-side perf \
             optimization (NOT landed in this iter)."
        );

        // (c) Qwen35 + Qwen3VL slot-aware fn surfaces still defined.
        for required in [
            "fn generate_qwen35_once_slot_aware",
            "fn generate_stream_qwen35_once_extended_slot_aware",
            "fn embed_qwen35_slot_aware",
            "fn generate_qwen35_once_with_soft_tokens_slot_aware",
            "fn generate_qwen35_once_with_soft_tokens_and_deepstack_slot_aware",
        ] {
            let src_q = include_str!("engine_qwen35.rs");
            assert!(
                src_q.contains(required),
                "H211 FALSIFIED: required Qwen35 slot-aware fn \
                 `{required}` is missing from engine_qwen35.rs.  \
                 iter-LCP / iter-G must not delete any slot-aware fn."
            );
        }
    }

    /// **H212 (skip-mode)** — production-default sampling + logprobs
    /// paths UNCHANGED by iter-G.  iter-G touches ONLY the greedy fast-
    /// path branches; sampling + logprobs branches still use
    /// `forward_gpu_last_logits` + `sample_logits_qwen35[_with_logprob]`.
    #[test]
    fn h212_sampling_and_logprobs_paths_unchanged_by_iter_g() {
        let src = include_str!("engine_qwen35.rs");

        // (a) `sample_logits_qwen35` + `sample_logits_qwen35_with_logprob`
        // are still called in slot-aware fns.  Each slot-aware fn's
        // non-greedy branch uses one of these.
        let sample_count = src.matches("sample_logits_qwen35(").count()
            + src.matches("sample_logits_qwen35_with_logprob(").count();
        assert!(
            sample_count >= 4,
            "H212 FALSIFIED: `sample_logits_qwen35` + \
             `sample_logits_qwen35_with_logprob` total call count \
             {sample_count} < 4.  iter-G must NOT touch the sampling \
             branches — the sampling + logprobs paths require full \
             logits CPU-side, not the GPU-argmax fast-path."
        );

        // (b) `forward_gpu_last_logits` is still called from slot-aware
        // fns (the non-greedy branches).  We pin ≥6 call sites total
        // (each slot-aware fn has prefill + decode-sampling +
        // decode-logprobs branches that go through forward_gpu_last_
        // logits).
        let last_logits_count = src.matches(".forward_gpu_last_logits(").count();
        assert!(
            last_logits_count >= 6,
            "H212 FALSIFIED: `forward_gpu_last_logits` call count \
             {last_logits_count} < 6.  iter-G must NOT route ALL \
             decode branches through `forward_gpu_greedy` — sampling + \
             logprobs branches must stay on `forward_gpu_last_logits`."
        );

        // (c) ADR-040 §6.1.50 closure block exists (will be added when
        // ADR-040 is updated by this iter).
        let adr =
            crate::serve::api::engine::adr040_history_doc() /* iter-230 A1: §6.1.x moved to history (aeb6e87c) */;
        assert!(
            adr.contains("### 6.1.50"),
            "H212 FALSIFIED: ADR-040 §6.1.50 closure block not found. \
             iter-LCP + iter-G + iter-2D-lcp joint closure must add \
             the §6.1.50 closure block to ADR-040."
        );
        // The closure block must name all 3 iters.
        let section_idx = adr
            .find("### 6.1.50")
            .expect("H212 (c): just asserted §6.1.50 present");
        let section_end_rel = adr[section_idx + 10..]
            .find("\n### ")
            .unwrap_or(adr.len() - section_idx - 10);
        let section_window = &adr[section_idx..(section_idx + 10 + section_end_rel).min(adr.len())];
        for required_label in [
            "iter-C2d-cont-kernel-iter-LCP",
            "iter-C2d-cont-kernel-iter-G",
            "iter-B4c-kernel-iter-2D-lcp",
        ] {
            assert!(
                section_window.contains(required_label),
                "H212 FALSIFIED: §6.1.50 closure block does NOT name \
                 `{required_label}`.  The joint closure must enumerate \
                 all 3 iters."
            );
        }
    }
}

// ---------------------------------------------------------------------------
// ADR-040 Phase C iter-C2e (2026-05-30) — Qwen3-VL SlotAware engine
// activation via Path B typed clamp.
//
// Direct mirror of C2c §6.1.21 (Gemma 4) + C2d §6.1.22 (Qwen35) for the
// Qwen3-VL text-LM family.  Pre-C2e the `Engine::spawn_with_mode(..,
// EngineMode::SlotAware { max_slots: N })` arm for Qwen3-VL returned
// `Err(EngineSpawnError::ModeNotYetWired { iter_required: "C2e (...)" })`.
// Iter-C2e flips that arm to `Ok(Engine)` via Path B:
//
//  1) Witness-only provisioning (no per-layer KV alloc): Qwen3-VL today
//     runs the iter-9b naive O(N²) re-prefill loop with no persistent
//     KV cache; the real cache is upstream-blocked on iter-228a (501
//     sentinel).  Method `Qwen3VlTextLoadedModel::
//     provision_multi_seq_kv_for_slot_aware(max_slots)` is a witness
//     scalar setter.
//  2) Four worker-arm typed clamps (Generate / GenerateStream / Embed /
//     GenerateWithSoftTokens) surface `MultiSeqError::Capability
//     Unsupported` at SlotId(N>0) with label naming `iter-C2e-cont per
//     ADR-040 §6.1.52` (post iter-228a worker-hot-path lift) AND
//     `iter-228a` (the upstream-blocker for the persistent KV cache
//     itself).
//
// New typed-error variant:
//
//   EngineSpawnError::Qwen3VLSlotAwareProvisionFailed { max_slots, cause }
//
// (mirror of `Gemma4SlotAwareProvisionFailed` + `Qwen35SlotAwareProvision
// Failed` shapes).
//
// Tests below pin H218-H223 per the iter-C2e spec.
#[cfg(test)]
mod adr040_phase_c_iter_c2e_qwen3vl_slot_aware_tests {
    use super::*;

    // ── Helper: snip worker_run body via the same shape used by C2d-cont /
    // B4c-kernel test modules. ──
    fn worker_run_body(src: &str) -> &str {
        let body_start = src
            .find("fn worker_run(")
            .expect("C2e: worker_run entry not found");
        let body_after = &src[body_start..];
        let body_end_off = body_after
            .find("\n// The worker thread for `LoadedModel::Qwen35` returns a sentinel error")
            .or_else(|| body_after.find("\n/// Worker-thread entry point"))
            .unwrap_or(body_after.len().min(200_000));
        &body_after[..body_end_off]
    }

    /// **H218 (skip-mode)** — post-C2e, Qwen3VL SlotAware spawn arm
    /// no longer returns `ModeNotYetWired`. Mirror of C2c H21 + C2d H26
    /// for the Qwen3-VL family.
    ///
    /// Skip-mode source-grep: we can't construct a real
    /// `LoadedModel::Qwen3VlText` without a GGUF, so we verify the
    /// spawn arm body in source. The pre-C2e `LoadedModel::Qwen3VlText(_)
    /// => Err(EngineSpawnError::ModeNotYetWired { iter_landed: "C2c", ...})`
    /// is REPLACED by the new arm body that calls
    /// `provision_multi_seq_kv_for_slot_aware` and returns
    /// `Ok(spawn_inner_with_slot_aware(...))`.
    #[test]
    fn h218_qwen3vl_spawn_arm_no_longer_returns_mode_not_yet_wired() {
        let src = include_str!("engine.rs");
        // Slice the spawn_with_mode body so the negative pin doesn't
        // accidentally match this test's OWN assert message
        // (include_str! pulls the entire file including these
        // assertions; the negative grep must be scoped to the actual
        // spawn_with_mode body only).
        let body_start = src
            .find("pub fn spawn_with_mode(")
            .expect("H218: spawn_with_mode entry not found");
        let body_end = body_start
            + src[body_start..]
                .find("    fn spawn_inner_with_slot_aware")
                .expect("H218: spawn_inner_with_slot_aware sibling not found");
        let body = &src[body_start..body_end];
        // Old pre-C2e marker must be GONE in the spawn_with_mode body.
        assert!(
            !body.contains("LoadedModel::Qwen3VlText(_) => Err(EngineSpawnError::ModeNotYetWired"),
            "H218 FALSIFIED: pre-C2e Qwen3VL ModeNotYetWired arm body \
             still present in spawn_with_mode body. iter-C2e must REPLACE \
             the ModeNotYetWired return with the witness-provisioning \
             arm body."
        );
        // New post-C2e arm body must be present (matches the C2d arm
        // pattern: `LoadedModel::Qwen3VlText(mut v) => {`).
        assert!(
            body.contains("LoadedModel::Qwen3VlText(mut v) => {"),
            "H218 FALSIFIED: post-C2e Qwen3VL SlotAware arm body marker \
             `LoadedModel::Qwen3VlText(mut v) => {{` not found in \
             spawn_with_mode body. The C2e spawn-arm flip must mirror \
             C2d's `LoadedModel::Qwen35(mut q)` shape."
        );
        // The new arm must call the provisioner + delegate to the
        // shared `spawn_inner_with_slot_aware` helper (mirror of C2c
        // + C2d).
        assert!(
            body.contains("v.provision_multi_seq_kv_for_slot_aware(max_slots)"),
            "H218 FALSIFIED: Qwen3VL spawn arm does NOT call \
             `provision_multi_seq_kv_for_slot_aware`. The C2e arm must \
             invoke the witness provisioner to mirror C2c+C2d shape."
        );
        assert!(
            body.contains("LoadedModel::Qwen3VlText(v)"),
            "H218 FALSIFIED: Qwen3VL spawn arm does not re-wrap the \
             loaded model as `LoadedModel::Qwen3VlText(v)` for \
             `spawn_inner_with_slot_aware`. C2e arm body shape broken."
        );
    }

    /// **H219 (skip-mode)** — Qwen3VL multi-seq KV "scaffold" provisioned
    /// at spawn is the witness scalar `slot_aware_max_slots: Option<u32>`
    /// per the iter-228a-blocked KV regime (no real per-layer cache
    /// yet; the persistent cache lands at iter-C2e-cont post iter-228a).
    ///
    /// Skip-mode source-grep on `engine_qwen3vl.rs`: the field is
    /// declared on `Qwen3VlTextLoadedModel`, initialized to `None` in
    /// `load`, and set to `Some(max_slots)` by
    /// `provision_multi_seq_kv_for_slot_aware`.
    #[test]
    fn h219_qwen3vl_witness_scalar_provisioned_at_spawn() {
        let src = include_str!("engine_qwen3vl.rs");
        // Field declared.
        assert!(
            src.contains("pub slot_aware_max_slots: Option<u32>"),
            "H219 FALSIFIED: `Qwen3VlTextLoadedModel.slot_aware_max_slots: \
             Option<u32>` field missing. C2e witness-scalar provisioning \
             requires this field per §6.1.52."
        );
        // Initialized to None in load.
        assert!(
            src.contains("slot_aware_max_slots: None,"),
            "H219 FALSIFIED: `load()` does NOT initialize \
             `slot_aware_max_slots: None`. The witness must default to \
             None so SerialFifo dispatch leaves it untouched (H222 \
             byte-equivalence pin)."
        );
        // Provision method exists + sets `Some(max_slots)`.
        assert!(
            src.contains("pub fn provision_multi_seq_kv_for_slot_aware"),
            "H219 FALSIFIED: `provision_multi_seq_kv_for_slot_aware` \
             method not declared on `Qwen3VlTextLoadedModel`. C2e \
             spawn-arm flip requires this method."
        );
        assert!(
            src.contains("self.slot_aware_max_slots = Some(max_slots);"),
            "H219 FALSIFIED: provision method does NOT set \
             `slot_aware_max_slots = Some(max_slots)`. Witness scalar \
             contract broken."
        );
        // The max_slots == 0 defense-in-depth bail is present.
        assert!(
            src.contains("ADR-040 C2e: provision_multi_seq_kv_for_slot_aware called with"),
            "H219 FALSIFIED: provision method does NOT contain the \
             ADR-040 C2e max_slots==0 anyhow::bail defense-in-depth. \
             Mirror of C2c/C2d provision-method invariants."
        );
    }

    /// **H220 (skip-mode)** — each of the four worker arms (Generate /
    /// GenerateStream / Embed / GenerateWithSoftTokens) carries the
    /// Qwen3VL `slot_id != SlotId(0)` typed clamp.
    ///
    /// The clamp label must name `iter-C2e-cont per ADR-040 §6.1.52`
    /// (the forward-pointer to the worker-hot-path lift) AND `iter-228a`
    /// (the upstream-blocker for the persistent KV cache itself, per
    /// §6.1.22's C2e cite). Operator-grep'able.
    #[test]
    fn h220_qwen3vl_worker_arms_typed_clamp_at_slot_n_gt_0() {
        let src = include_str!("engine.rs");
        let body = worker_run_body(src);
        // Four occurrences of the Qwen3VL clamp predicate (one per
        // Request variant).
        let clamp_predicate =
            "matches!(loaded, LoadedModel::Qwen3VlText(_)) && handle.slot_id != SlotId(0)";
        let n_occurrences = body.matches(clamp_predicate).count();
        assert!(
            n_occurrences >= 4,
            "H220 FALSIFIED: worker_run body contains {n_occurrences} \
             Qwen3VL `slot_id != SlotId(0)` clamp predicates; expected \
             at least 4 (one per Request variant: Generate, \
             GenerateStream, Embed, GenerateWithSoftTokens)."
        );
        // The clamp label inside the typed-error message names both
        // `iter-C2e-cont per ADR-040 §6.1.52` AND `iter-228a`.
        assert!(
            body.contains("iter-C2e-cont per ADR-040 §6.1.52"),
            "H220 FALSIFIED: Qwen3VL clamp label does NOT contain \
             `iter-C2e-cont per ADR-040 §6.1.52` — the forward-pointer \
             to the worker-hot-path lift iter. Operator log greps + \
             future-iter implementers depend on this literal cite."
        );
        assert!(
            body.contains("iter-228a"),
            "H220 FALSIFIED: Qwen3VL clamp label does NOT contain \
             `iter-228a` — the upstream blocker for the Qwen3-VL \
             forward path past the 501 sentinel. Operator triage needs \
             this cite to disambiguate from the C2d-cont label shape."
        );
        // The four arm-specific sub-labels per the §6.1.52 closure
        // discipline.
        for sublabel in [
            "qwen3vl-generate-slot-N",
            "qwen3vl-generate-stream-slot-N",
            "qwen3vl-embed-slot-N",
            "qwen3vl-generate-with-soft-tokens-slot-N",
        ] {
            assert!(
                body.contains(sublabel),
                "H220 FALSIFIED: Qwen3VL clamp sublabel `{sublabel}` \
                 missing from worker_run.  The four arm-specific cites \
                 mirror C2c §6.1.21's `gemma4-*-slot-N` per-arm labels."
            );
        }
    }

    /// **H221 (sibling discipline)** — Qwen35 + Gemma 4 surfaces are
    /// UNCHANGED by iter-C2e (only the Qwen3VL arm is modified).
    ///
    /// Source-grep across `engine.rs`:
    /// - Both `Gemma4SlotAwareProvisionFailed` and
    ///   `Qwen35SlotAwareProvisionFailed` typed-error variants still
    ///   declared.
    /// - C2c `LoadedModel::Gemma(mut g) => {` arm body still present.
    /// - C2d `LoadedModel::Qwen35(mut q) => {` arm body still present.
    /// - All four Gemma 4 worker-arm lifts still called via their
    ///   slot-aware orchestrator fns (B4c-kernel iter-1/3/4/5).
    /// - All four Qwen35 worker-arm lifts still called via their
    ///   slot-aware orchestrator fns (C2d-cont-kernel iter-1/2/3/4).
    #[test]
    fn h221_qwen35_and_gemma4_surfaces_unchanged_by_c2e() {
        let src = include_str!("engine.rs");
        // Typed-error siblings still declared.
        assert!(
            src.contains("Gemma4SlotAwareProvisionFailed"),
            "H221 FALSIFIED: `Gemma4SlotAwareProvisionFailed` removed \
             by C2e. C2e must NOT touch the Gemma 4 typed-error surface."
        );
        assert!(
            src.contains("Qwen35SlotAwareProvisionFailed"),
            "H221 FALSIFIED: `Qwen35SlotAwareProvisionFailed` removed \
             by C2e. C2e must NOT touch the Qwen35 typed-error surface."
        );
        assert!(
            src.contains("Gemma4HybridSlotAwareProvisionFailed"),
            "H221 FALSIFIED: `Gemma4HybridSlotAwareProvisionFailed` \
             removed by C2e. C2e must NOT touch the iter-C2c-cont \
             Gemma 4 hybrid-scaffold typed-error surface."
        );
        // C2c + C2d spawn-arm bodies still present.
        assert!(
            src.contains("LoadedModel::Gemma(mut g) => {"),
            "H221 FALSIFIED: C2c Gemma 4 spawn-arm body marker missing."
        );
        assert!(
            src.contains("LoadedModel::Qwen35(mut q) => {"),
            "H221 FALSIFIED: C2d Qwen35 spawn-arm body marker missing."
        );
        // Gemma 4 worker-arm lift fns still called.
        for lift_fn in [
            "generate_gemma4_once_slot_aware(",
            "generate_stream_gemma4_once_slot_aware(",
            "embed_gemma4_slot_aware(",
            "generate_gemma4_once_with_soft_tokens_slot_aware(",
        ] {
            assert!(
                src.contains(lift_fn),
                "H221 FALSIFIED: Gemma 4 lift fn `{lift_fn}` is NOT \
                 called from worker_run. C2e must NOT regress any \
                 Gemma 4 worker-arm lift (§6.1.31/35/36/37)."
            );
        }
        // Qwen35 worker-arm lift fns still called.
        for lift_fn in [
            "super::engine_qwen35::generate_qwen35_once_slot_aware(",
            "super::engine_qwen35::generate_stream_qwen35_once_extended_slot_aware(",
            "super::engine_qwen35::embed_qwen35_slot_aware(",
            "super::engine_qwen35::generate_qwen35_once_with_soft_tokens_slot_aware(",
        ] {
            assert!(
                src.contains(lift_fn),
                "H221 FALSIFIED: Qwen35 lift fn `{lift_fn}` is NOT \
                 called from worker_run. C2e must NOT regress any \
                 Qwen35 worker-arm lift (§6.1.27/28/29/30)."
            );
        }
    }

    /// **H222 (SerialFifo byte-equivalence pin)** — Qwen3VL SerialFifo
    /// path is UNCHANGED by C2e. The pre-C2e EngineMode::SerialFifo
    /// dispatch did NOT call `provision_multi_seq_kv_for_slot_aware`,
    /// and post-C2e MUST still not call it (otherwise SerialFifo would
    /// gain a per-spawn witness write that breaks byte-equivalence).
    /// Mirror of C2c's H23 + C2d's H28 source-grep regression-pin
    /// pattern.
    #[test]
    fn h222_serial_fifo_qwen3vl_does_not_provision_multi_seq_kv() {
        let src = include_str!("engine.rs");
        let body_start = src
            .find("pub fn spawn_with_mode(")
            .expect("H222: spawn_with_mode entry not found");
        let body_end = body_start
            + src[body_start..]
                .find("    fn spawn_inner_with_slot_aware")
                .expect("H222: spawn_inner_with_slot_aware sibling not found")
            + "    fn spawn_inner_with_slot_aware".len();
        let body = &src[body_start..body_end];
        let serial_fifo_idx = body
            .find("EngineMode::SerialFifo")
            .expect("H222: SerialFifo arm not found in spawn_with_mode");
        let slot_aware_idx = body
            .find("EngineMode::SlotAware")
            .expect("H222: SlotAware arm not found in spawn_with_mode");
        assert!(
            serial_fifo_idx < slot_aware_idx,
            "H222 sanity: dispatch table orders SerialFifo before SlotAware"
        );
        let serial_fifo_arm = &body[serial_fifo_idx..slot_aware_idx];
        assert!(
            !serial_fifo_arm.contains("provision_multi_seq_kv_for_slot_aware"),
            "H222 FALSIFIED: post-C2e SerialFifo arm now calls \
             provision_multi_seq_kv_for_slot_aware — byte-equivalence \
             with pre-C2e behavior broken (the Qwen3VL provisioner is \
             a witness-only setter today but on iter-228a will alloc \
             real KV — SerialFifo must never engage either path)."
        );
        // Also: the iter-228a 501 sentinel routing in the four worker
        // arms is preserved verbatim — the C2e clamp short-circuits
        // BEFORE the sentinel dispatch at SlotId(N>0), but SlotId(0)
        // still hits the existing sentinel routing for the
        // non-Generate-arm cases (Embed / GenerateWithSoftTokens have
        // soft-token guards). Pin via source-grep on the existing
        // sentinel call site (engine.rs:~5697+ etc.).
        assert!(
            src.contains("qwen3vl_text_forward_pending_err"),
            "H222 FALSIFIED: the iter-228a 501 sentinel routing \
             (`qwen3vl_text_forward_pending_err`) is missing from \
             engine.rs. C2e must NOT touch the iter-228a sentinel path."
        );
    }

    /// **H223 (typed-error variant exists)** — the new
    /// `EngineSpawnError::Qwen3VLSlotAwareProvisionFailed { max_slots,
    /// cause }` variant exists with the expected shape (mirror of
    /// `Gemma4SlotAwareProvisionFailed` + `Qwen35SlotAwareProvision
    /// Failed`).
    #[test]
    fn h223_qwen3vl_slot_aware_provision_failed_variant_exists_with_max_slots_and_cause() {
        let err = EngineSpawnError::Qwen3VLSlotAwareProvisionFailed {
            max_slots: 4,
            cause: "synthetic test cause".to_string(),
        };
        let msg = format!("{}", err);
        assert!(
            msg.contains("Qwen3-VL") || msg.contains("qwen3vl") || msg.contains("C2e"),
            "H223 FALSIFIED: post-C2e Qwen3VLSlotAwareProvisionFailed \
             Display must identify the failing arch + iter. Got: {msg}"
        );
        assert!(
            msg.contains("4"),
            "H223 sanity: Qwen3VLSlotAwareProvisionFailed Display must \
             include max_slots value. Got: {msg}"
        );
        // Pin destructuring shape (catches future field rename / removal).
        match err {
            EngineSpawnError::Qwen3VLSlotAwareProvisionFailed { max_slots, cause } => {
                assert_eq!(max_slots, 4, "H223: max_slots field roundtrips");
                assert_eq!(cause, "synthetic test cause", "H223: cause roundtrips");
            }
            _ => panic!("H223 FALSIFIED: variant structure changed unexpectedly"),
        }
    }

    /// **H223-cont (ADR §6.1.52 closure block pin)** — the C2e
    /// closure block exists in ADR-040 and names the four arm-specific
    /// cite labels + the iter-C2e-cont follow-up + the iter-228a
    /// upstream-blocker.
    #[test]
    fn h223_cont_adr_section_6_1_52_closure_block_named() {
        let adr = crate::serve::api::engine::adr040_history_doc() /* iter-230 A1: §6.1.x moved to history (aeb6e87c) */;
        assert!(
            adr.contains("### 6.1.52"),
            "H223-cont FALSIFIED: ADR-040 §6.1.52 closure block not \
             found. iter-C2e SHIPPED must add the §6.1.52 closure block \
             per the §6.1.N-per-iter discipline."
        );
        let section_idx = adr
            .find("### 6.1.52")
            .expect("H223-cont (a): §6.1.52 just asserted present");
        let section_end_rel = adr[section_idx + 10..]
            .find("\n### ")
            .unwrap_or(adr.len() - section_idx - 10);
        let section_window = &adr[section_idx..(section_idx + 10 + section_end_rel).min(adr.len())];
        for required_label in ["iter-C2e", "Qwen3-VL", "iter-C2e-cont", "iter-228a"] {
            assert!(
                section_window.contains(required_label),
                "H223-cont FALSIFIED: §6.1.52 closure block does NOT \
                 name `{required_label}`.  The C2e closure must \
                 enumerate the iter + arch + follow-up + upstream-blocker."
            );
        }
    }

    // ──────────────────────────────────────────────────────────────────
    // ADR-040 §6.1.55 FINAL CLOSURE BUNDLE (2026-05-30) —
    // H236 / H237 / H238 / H239 / H240 source-grep pins for the 5
    // surviving deferrals SHIPPED structurally as one bundle.
    //
    // - H236: iter-A4-cont-moe-validation env-gated harness scaffold.
    // - H237: iter-C2e-cont structural worker hot path lift.
    // - H238: ADR-040 §6.1.55 closure block exists + names the bundle.
    // - H239: SerialFifo byte-equivalence preserved across all 5 lifts.
    // - H240: Qwen35 / Gemma 4 / non-A4 + non-spec-decode surfaces UNCHANGED.
    // ──────────────────────────────────────────────────────────────────

    /// **H236** — `iter-A4-cont-moe-validation` env-gated harness
    /// scaffold lives at `tests/continuous_batching_throughput.rs` per
    /// the dossier §6 typed-deferral name.  Source-grep pin only — no
    /// hardware engagement.  Operator-runnable via
    /// `HF2Q_A4_MOE_AB_VALIDATION_E2E=1` + `HF2Q_CB_THROUGHPUT_MODEL`.
    #[test]
    fn h236_iter_a4_cont_moe_validation_env_gated_harness_exists() {
        let bench_src = include_str!("../../../tests/continuous_batching_throughput.rs");
        assert!(
            bench_src.contains("HF2Q_A4_MOE_AB_VALIDATION_E2E"),
            "H236 FALSIFIED: iter-A4-cont-moe-validation harness MUST \
             gate on HF2Q_A4_MOE_AB_VALIDATION_E2E env per the dossier \
             §6 typed-deferral name + the D3 operator-runnable mirror."
        );
        assert!(
            bench_src.contains("a4_moe_validation_qwen36_a3b_a_b_n_1_2_4_8"),
            "H236 FALSIFIED: iter-A4-cont-moe-validation harness test \
             name MUST be `a4_moe_validation_qwen36_a3b_a_b_n_1_2_4_8` \
             so operators can target it by name."
        );
        assert!(
            bench_src.contains("iter-A4-cont-moe-validation"),
            "H236 FALSIFIED: harness MUST carry the `iter-A4-cont-moe-validation` \
             cite for operator-grep + ADR §6.1.55 cross-reference."
        );
        // Acceptance-rate dimension cell also lives at the bench file
        // — pin the iter-A4-cont-inflection-bench scaffold here for
        // colocation with the MoE-validation harness.
        assert!(
            bench_src.contains("HF2Q_A4_INFLECTION_BENCH"),
            "H236 (companion) FALSIFIED: iter-A4-cont-inflection-bench \
             harness MUST gate on HF2Q_A4_INFLECTION_BENCH env."
        );
        assert!(
            bench_src.contains("AcceptanceCell"),
            "H236 (companion) FALSIFIED: AcceptanceCell carrier MUST exist \
             at the bench file per dossier §5 + §6.1.55."
        );
        assert!(
            bench_src.contains("render_acceptance_report"),
            "H236 (companion) FALSIFIED: render_acceptance_report helper \
             MUST exist for operator-readable plotting."
        );
    }

    /// **H237** — iter-C2e-cont structural worker hot path lift.
    /// The four worker-arm clamps now call the
    /// [`crate::serve::api::engine_qwen3vl::Qwen3VlTextLoadedModel::
    /// handle_qwen3vl_slot_aware_n_gt_0_sentinel`] helper instead of
    /// emitting inline `anyhow!` literals.  Witness take/restore is
    /// the structural lift step.  Sentinel propagation preserved
    /// verbatim (H240 + H222 cross-pin).
    #[test]
    fn h237_iter_c2e_cont_structural_worker_hot_path_lift_via_helper() {
        let engine_src = include_str!("engine.rs");
        // The helper is named at the worker hot path (called from
        // each of the four worker arms).
        let n_helper_calls = engine_src
            .matches("handle_qwen3vl_slot_aware_n_gt_0_sentinel")
            .count();
        assert!(
            n_helper_calls >= 4,
            "H237 FALSIFIED: helper `handle_qwen3vl_slot_aware_n_gt_0_sentinel` \
             called {n_helper_calls} times; expected at least 4 (one per \
             worker arm: Generate / GenerateStream / Embed / GenerateWithSoftTokens)."
        );
        // The iter-C2e-cont cite is named at each of the 4 worker arms
        // for forward-pointer to §6.1.55.
        let n_cont_cites = engine_src
            .matches("iter-C2e-cont per ADR-040 §6.1.55")
            .count();
        assert!(
            n_cont_cites >= 4,
            "H237 FALSIFIED: `iter-C2e-cont per ADR-040 §6.1.55` cite \
             appears {n_cont_cites} times; expected at least 4 (one per \
             worker arm for operator-grep)."
        );
        // The helper itself lives at engine_qwen3vl.rs.
        let qwen3vl_src = include_str!("engine_qwen3vl.rs");
        assert!(
            qwen3vl_src.contains("pub fn handle_qwen3vl_slot_aware_n_gt_0_sentinel"),
            "H237 FALSIFIED: helper declaration missing from engine_qwen3vl.rs."
        );
        // Take/restore witness discipline is the structural lift step.
        assert!(
            qwen3vl_src.contains("self.slot_aware_max_slots.take()"),
            "H237 FALSIFIED: helper MUST `take()` the slot_aware_max_slots \
             witness scalar — this is the structural-lift mirror of \
             Qwen35 / Gemma 4 `persistent_kv_cache.take()` discipline."
        );
        assert!(
            qwen3vl_src.contains("self.slot_aware_max_slots = witness"),
            "H237 FALSIFIED: helper MUST restore the witness post-sentinel \
             — preserves the spawn-time invariant `slot_aware_max_slots.is_some()` \
             for SlotAware engines across the worker arm boundary."
        );
        // Sentinel delegation: the helper MUST call the iter-228a
        // 501 sentinel verbatim.  This is the H240 propagation pin.
        assert!(
            qwen3vl_src.contains("qwen3vl_text_forward_pending_err"),
            "H237 FALSIFIED: helper MUST delegate to the iter-228a 501 \
             sentinel (`qwen3vl_text_forward_pending_err`) — sentinel \
             propagation contract preserved verbatim."
        );
    }

    /// **H238** — ADR-040 §6.1.55 closure block exists and names
    /// "ADR-040 FULL IMPLEMENTATION CLOSURE" with all five surviving
    /// deferrals SHIPPED structurally.
    #[test]
    fn h238_adr_section_6_1_55_full_implementation_closure_block() {
        let adr = crate::serve::api::engine::adr040_history_doc() /* iter-230 A1: §6.1.x moved to history (aeb6e87c) */;
        assert!(
            adr.contains("### 6.1.55"),
            "H238 FALSIFIED: ADR-040 §6.1.55 closure block not found. \
             The final-bundle iter SHIPPED must add a §6.1.55 closure \
             block per the §6.1.N-per-iter discipline."
        );
        assert!(
            adr.contains("ADR-040 FULL IMPLEMENTATION CLOSURE"),
            "H238 FALSIFIED: §6.1.55 closure block MUST carry the title \
             `ADR-040 FULL IMPLEMENTATION CLOSURE` so operator searches \
             land directly on the final-bundle closure."
        );
        let section_idx = adr
            .find("### 6.1.55")
            .expect("H238 (a): §6.1.55 just asserted present");
        let section_end_rel = adr[section_idx + 10..]
            .find("\n### ")
            .unwrap_or(adr.len() - section_idx - 10);
        let section_window = &adr[section_idx..(section_idx + 10 + section_end_rel).min(adr.len())];
        // Names the 5 surviving deferrals.
        for required_label in [
            "iter-A4-cont-acceptance-telemetry",
            "iter-A4-cont-inflection-bench",
            "iter-A4-cont-drafter-dispatcher",
            "iter-A4-cont-moe-validation",
            "iter-C2e-cont",
        ] {
            assert!(
                section_window.contains(required_label),
                "H238 FALSIFIED: §6.1.55 closure block does NOT name \
                 `{required_label}`.  The final-bundle closure must \
                 enumerate ALL 5 surviving deferrals SHIPPED structurally."
            );
        }
    }

    /// **H239 (SerialFifo byte-equivalence pin)** — the SerialFifo
    /// dispatch path is UNCHANGED by §6.1.55.  None of the 5 lifts
    /// add a worker-arm path on SerialFifo at SlotId(0).
    ///
    /// Source-grep pins:
    /// - SerialFifo arm of `spawn_with_mode` does NOT call any of the
    ///   new iter-A4-cont* helpers OR the new iter-C2e-cont helper.
    /// - Worker arm clamps still gated on
    ///   `handle.slot_id != SlotId(0)` — SerialFifo always emits
    ///   SlotId(0) (FifoSchedulerAdapter invariant; H51 cross-pin).
    /// - The DrafterKvCacheVariant routing helper degrades to
    ///   SingleSeq at `max_slots <= 1` (pre-A4 byte-equivalent).
    #[test]
    fn h239_serial_fifo_byte_equivalence_preserved_across_all_5_lifts() {
        let engine_src = include_str!("engine.rs");
        let body_start = engine_src
            .find("pub fn spawn_with_mode(")
            .expect("H239: spawn_with_mode entry not found");
        let body_end = body_start
            + engine_src[body_start..]
                .find("    fn spawn_inner_with_slot_aware")
                .expect("H239: spawn_inner_with_slot_aware sibling not found");
        let body = &engine_src[body_start..body_end];
        let serial_fifo_idx = body
            .find("EngineMode::SerialFifo")
            .expect("H239: SerialFifo arm not found");
        let slot_aware_idx = body
            .find("EngineMode::SlotAware")
            .expect("H239: SlotAware arm not found");
        let serial_fifo_arm = &body[serial_fifo_idx..slot_aware_idx];
        // SerialFifo arm MUST NOT call any of the new helpers.
        for forbidden in [
            "handle_qwen3vl_slot_aware_n_gt_0_sentinel",
            "select_drafter_kv_variant_for_mode",
            "DrafterKvCacheVariant",
        ] {
            assert!(
                !serial_fifo_arm.contains(forbidden),
                "H239 FALSIFIED: SerialFifo arm contains `{forbidden}` — \
                 byte-equivalence with pre-§6.1.55 behaviour broken. \
                 The 5-deferral lifts MUST sit on the SlotAware-only \
                 dispatch surface."
            );
        }
        // Worker-arm clamp predicate is still `handle.slot_id != SlotId(0)`
        // (SerialFifo always hands out SlotId(0); H51 cross-pin).
        assert!(
            engine_src.contains("handle.slot_id != SlotId(0)"),
            "H239 FALSIFIED: worker-arm clamp predicate `handle.slot_id \
             != SlotId(0)` removed.  SerialFifo path requires this \
             predicate to short-circuit at SlotId(0) → fall through to \
             the existing single-seq dispatch (byte-equivalent)."
        );
        // DrafterKvCacheVariant routing degrades to SingleSeq at
        // max_slots <= 1 (pre-A4 byte-equivalent).
        let drafter_src = include_str!("../../inference/spec_decode/eagle3/kv_cache.rs");
        assert!(
            drafter_src.contains("if max_slots <= 1") || drafter_src.contains("max_slots == 1"),
            "H239 FALSIFIED: select_drafter_kv_variant_for_mode MUST \
             route max_slots <= 1 to SingleSeq (byte-equivalent fallback)."
        );
    }

    /// **H240** — Qwen35 / Gemma 4 / non-A4 + non-spec-decode surfaces
    /// UNCHANGED.  Sibling discipline preserved across §6.1.55.
    /// Source-grep across `engine.rs` + the eagle3 kv_cache:
    /// - The four Gemma 4 worker-arm lift fns still called.
    /// - The four Qwen35 worker-arm lift fns still called.
    /// - The Qwen35 `Qwen35SlotAwareProvisionFailed` typed-error still declared.
    /// - The Gemma 4 `Gemma4SlotAwareProvisionFailed` typed-error still declared.
    /// - The Qwen3VL `Qwen3VLSlotAwareProvisionFailed` typed-error still declared.
    /// - The iter-228a `qwen3vl_text_forward_pending_err` sentinel
    ///   routing preserved.
    /// - The LEGACY `DrafterKvCache` surface UNCHANGED.
    #[test]
    fn h240_qwen35_gemma4_non_a4_non_spec_decode_surfaces_unchanged() {
        let src = include_str!("engine.rs");
        // Typed-error siblings still declared.
        for variant in [
            "Gemma4SlotAwareProvisionFailed",
            "Qwen35SlotAwareProvisionFailed",
            "Qwen3VLSlotAwareProvisionFailed",
            "Gemma4HybridSlotAwareProvisionFailed",
            "SpecDecodeMaxSlotsAboveBatchedThreshold",
        ] {
            assert!(
                src.contains(variant),
                "H240 FALSIFIED: `{variant}` typed-error variant \
                 removed by §6.1.55. The final-bundle lift MUST NOT \
                 touch the per-arch typed-error surfaces."
            );
        }
        // Gemma 4 worker-arm lift fns still called.
        for lift_fn in [
            "generate_gemma4_once_slot_aware(",
            "generate_stream_gemma4_once_slot_aware(",
            "embed_gemma4_slot_aware(",
            "generate_gemma4_once_with_soft_tokens_slot_aware(",
        ] {
            assert!(
                src.contains(lift_fn),
                "H240 FALSIFIED: Gemma 4 lift fn `{lift_fn}` is NOT \
                 called from engine.rs. §6.1.55 must NOT regress any \
                 Gemma 4 worker-arm lift (§6.1.31/35/36/37)."
            );
        }
        // Qwen35 worker-arm lift fns still called.
        for lift_fn in [
            "super::engine_qwen35::generate_qwen35_once_slot_aware(",
            "super::engine_qwen35::generate_stream_qwen35_once_extended_slot_aware(",
            "super::engine_qwen35::embed_qwen35_slot_aware(",
            "super::engine_qwen35::generate_qwen35_once_with_soft_tokens_slot_aware(",
        ] {
            assert!(
                src.contains(lift_fn),
                "H240 FALSIFIED: Qwen35 lift fn `{lift_fn}` is NOT \
                 called from engine.rs. §6.1.55 must NOT regress any \
                 Qwen35 worker-arm lift (§6.1.27/28/29/30)."
            );
        }
        // iter-228a sentinel routing preserved verbatim.
        assert!(
            src.contains("qwen3vl_text_forward_pending_err"),
            "H240 FALSIFIED: iter-228a `qwen3vl_text_forward_pending_err` \
             sentinel routing removed. §6.1.55 iter-C2e-cont MUST \
             delegate to the upstream sentinel verbatim — sentinel \
             propagation preserved."
        );
        // The LEGACY DrafterKvCache surface UNCHANGED — no method
        // renames / signature flips at the iter-A4-cont-drafter-
        // dispatcher lift.
        let drafter_src = include_str!("../../inference/spec_decode/eagle3/kv_cache.rs");
        assert!(
            drafter_src.contains("pub struct DrafterKvCache "),
            "H240 FALSIFIED: legacy DrafterKvCache struct declaration \
             removed.  The dispatcher variant carrier is ADDITIVE per \
             dossier §5; the legacy single-seq surface is UNCHANGED."
        );
        assert!(
            drafter_src.contains("pub struct MultiSeqDrafterKvCache "),
            "H240 FALSIFIED: A4 iter-1 MultiSeqDrafterKvCache sibling \
             surface removed."
        );
    }
}