Module pp 

M2 pipeline-parallel N-stage runtime (generalizes the M1 2-stage seam).

Door: MEMRA_PP_STAGES=N (default OFF — unset/0/1 = no behavior change anywhere). Stage map: N stages over the trunk layers with N-1 cuts. MEMRA_PP_SPLITS=c1,..,cN-1 sets the cuts explicitly (strictly increasing, in (0, n_layers)); MEMRA_PP_SPLIT=<i> is the N=2 back-compat spelling; default = even split (cut s = s*n_layers/N). Placement: MEMRA_PP_DEVICES=d0,..,dN-1 maps stage s to device ds (default: all on the primary engine’s device).

M1 history (increments 1-2, merged + hardened on the 8x box 2026-08-02): seam + gate single-device; then real transport — per-stage streams/events, device placement, peer-copy boundary (M0: cudaMemcpyPeerAsync beats NCCL 2.8x at PP activation sizes), per-context PDL module caches, default-mempool peer grants. All five r3 gates PASS bit-identical (receipts ~/receipts/m1-pp2/ on ).

M2 increment 1 (this file): N-STAGE GENERALIZATION — Pp2Rt becomes PpNRt:

• stages: Vec of per-stage execution homes (device, context, stream, remote Engine);
• boundaries: N-1 boundary runtimes, each with TWO persistent double-buffered slots (ev_tx/ev_rx per slot) and its own overlap step counter; transport is selected PER BOUNDARY (dtod same-device / cudaMemcpyPeerAsync cross-device by default; opt-in MEMRA_PP_HOST_BOUNCE=1 uses pinned D2H + H2D instead);
• the default peer transport grants peer + default-mempool access between EVERY distinct pair of devices in use. Host bounce skips serving-time grants; its boot diagnostics transiently enable peer + pool access, then revoke the pool grants and disable peer access before proceeding. Sharded weights plus stage-local auxiliary buffers ensure that no peer read can bypass the bounced boundary.

M2 increment 2 (weight sharding): the loader uploads each stage’s layer range THROUGH that stage’s engine (layer_engine), so weights land on the device that runs them — the bring-up peer-read placement dies. output_norm + lm head load through the LAST stage’s engine; the embed table stays host-side with stage 0. Split-plane/f16 decode mirrors are built per layer through the owning stage’s engine too (the rp4 mirrors ARE the decode weights on the q8 path — leaving them on dev0 would fake the kill). Rollback seam: MEMRA_PP_SHARD=0 = M1 bring-up placement (all weights on primary, remote stages peer-read).

M2 increment 3 (deferred readback — the pipelining seed): PendingLogits — the eager decode arm can END a step without the logits D2H (decode_step_h_ppn_deferred): the logits stay device-resident with a completion event; wait() drains them through a DEDICATED readback stream (waits the event, copies, syncs) so tokens t+1.. keep enqueuing on the stage streams while token t drains. Per-token math is fully event-ordered (same slots, same ev_tx/ev_rx chain) — scheduling changes, math does not; the pipelined replay arm of ppn-gate proves bit-identity per step.

Ownership across a boundary (unchanged from M1):

• hidden state [n_embd] f32 is the ONLY tensor that crosses;
• KV/linear-attn cache entries are per-layer: stage s exclusively owns cache state for its layer range (and, under MEMRA_PP_DEVICES, allocates it on its device);
• position/rope state is the scalar cache.pos snapshot taken once per step; every stage uploads its own position buffer on its own stream (no cross-device position pointer);
• the embed table lives with stage 0, output_norm + lm head with the last stage.

THE MULTI-STREAM LAW (why this is safe with cudarc event tracking disabled): all cross-stage bytes flow through the persistent boundary slots, ordered by ev_tx/ev_rx; per-stage scratch is allocated AND freed on that stage’s stream (stream-ordered); the async mem pool runs with opportunistic reuse OFF + internal dependencies ON (memra-runtime), so a block freed on stream A and reused on stream B carries a driver-inserted dependency. Weights are load-time state no stage stream can precede, and the step’s terminal logits readback (sync D2H, or PendingLogits’ event-ordered readback stream) drains the last stage, whose TX-wait chain transitively drains all.

Scope: plain eager decode only (generic arm N-stage; gemma4 arm 2-stage). NOT wired: batch/dc/graph/spec loops and the gemma4-E4B eager arm.

CORRECTION (pp2-hardening 2026-08-06): this header used to add “(warn_unwired_once fires)” to that list, which was wrong. warn_unwired_once has exactly two call sites and BOTH are gemma4-specific (decode.rs, hybrid_forward.rs) — the batch/dc/graph/spec loops never warned. Worse, the batched loop did not merely run unsplit: it walked the whole trunk on the primary stream and, under a sharded cross-device placement, peer-read every remote stage’s weights each step — 28x slower at B=1 with all three decode-batch-gate gates PASSING (peer reads are byte-exact, so only perf broke). decode_step_batch now FAILS CLOSED in that regime via pp_sharded_cross_device() (MEMRA_PP_ALLOW_UNSPLIT_BATCH=1 = measurement override). “Unwired” for dc/graph/spec still means “runs unsplit, silently” — audit each before trusting it on a pair.

Structs

PeerProbeMetrics

PendingLogits

M2 increment 3: a step’s logits, still device-resident on the LAST stage. wait() orders the readback stream behind the step’s completion event, copies, and syncs — tokens enqueued after this step keep running on the stage streams while the caller drains token t. Dropping without waiting is safe (buffers free stream-ordered).

PpNRt

PpWalkLease

Opaque lifetime token for one complete PP boundary walk. Clones are allowed only through an explicit coordinator permit or the same-thread deferred enqueue window; the active generation clears when the final clone is dropped.

StageRt

One pipeline stage’s execution home: device, context, launch stream, and (for a stage remote to the primary engine’s device) a dedicated Engine in that device’s primary context (CUmodules are per-context).

Enums

DualPpMode

MEMRA_DUAL_PP three-state mode for the dual-active PP-2 batched decode path. Default ON (owner flip 2026-08-11) after the box1 PRO-pair re-gate: correctness bit-identity B=1..5, servestress no-thrash, 10-boot soak 929/929 golden matches with 0 slot collisions across 9123 pairs (research/dualpp2-20260811/RESULTS-regate.md), plus the dualpp1 c>=8 interleaved perf floor (+20.753% minimum, research/dualpp1-20260811/RESULTS.md).

PeerProbeStartupPolicy

RuntimePeerProbeStatus

Constants

DUAL_PP_HOST_BOUNCE_REFUSAL

DUAL_PP_SINGLE_SLOT_REFUSAL

Binding-amendment refusal text. The negative gate quotes this exact line and requires the decode call to return before producing a token or advancing a cache.

DUAL_PP_STAGE_NAMES

PEER_PROBE_REQUIRED_REFUSAL

PEER_RUNTIME_PROBE_CYCLE_COPIES

One complete runtime width rotation. The maximum-chunk rung runs once per cycle.

PEER_RUNTIME_PROBE_DEFERRAL_BOUND_INTERVALS

Consecutive runnable probe intervals that may be blocked by live speculative UVA state before integrity coverage becomes explicitly degraded. Four intervals are one full width rotation.

PEER_RUNTIME_PROBE_INTERVAL_COPIES

Native cross-device boundary copies between low-frequency runtime integrity probes. Fixed rather than operator-tunable: this is a safety gate, not a performance experiment.

PP_WAVE_MAX_STAGES

Largest PP wavefront admitted by the RTX PRO 6000 product shape. The underlying placement runtime remains N-stage, but the serving wave scheduler is deliberately bounded to the 2–4 card surface that has a concrete qualification plan.

PP_WAVE_W4A16_BF16_REFUSAL

The PP3/PP4 scheduler decomposes one serving batch into narrower stage waves. A preserved-BF16 W4A16 artifact therefore needs the row-wise BF16 matvec program: the default f32-expanded cuBLAS path is batch-width dependent, and HY3 B=4/B=8 changed every logit row when wave cells narrowed to B=1/B=2. MEMRA_BF16_MMV=1 keeps the same checkpoint BF16 values and runs one deterministic per-row reduction program at every width.

Statics

PRIME_PIPE_OVERLAPS

PIPELINE-LIVENESS COUNTER: bumped only when a second PP-2 prime stage enters its layer walker while the other stage’s walker is still active. Step’s per-layer router readback synchronizes the host, so enqueue order alone is not liveness: a single host thread can call stage 0(N+1) before the stage-1 epilogue and still serialize all trunk computation.

PRIME_SPLIT_CHUNKS

SPLIT-LIVENESS COUNTER for the prime stage split: bumped ONCE per prime chunk that actually executed the per-stage walk. The prime-split-gate requires this to ADVANCE during its split arm — bit-identity of two identical UNSPLIT walks is vacuous, so a gate that only compared bits would go green while the walker doesn’t exist. With the counter, the gate is RED until the walker lands (the tickinv35 pattern: the gate exists and fails before the mechanism does). Relaxed ordering: single-threaded host issue, count-only.

STEP35_PRIME_BATCHES

Step35 cross-request prime liveness counters (lane/cx-prime-batch, 2026-08-08). The exactness gate requires BOTH to advance: a successful step35 batch alone is not sufficient under PP-N if it walked the whole sharded trunk on one stream.

STEP35_PRIME_BATCH_SPLITS

Functions

batch_pp_on

MEMRA_BATCH_PP=0: rollback/A-B seam for the BATCHED stage split (pp2-batch 2026-08-06). Default ON — with the ppN door open the batched decode step takes its own stage split (decode_step_batch_ppn) exactly as the eager step does. Setting 0 sends the batched path back through the unsplit body, which under a sharded cross-device placement is then caught by refuse_unsplit_if_remote (the 28x peer-read regime) rather than run silently. Exists so the bit-identity gate can A/B split vs unsplit IN ONE PROCESS against the same loaded weights — read per step, never memoized, for that reason.

clear_runtime_peer_probe_integrity_degraded

A completed native probe or validated transport failover restores an explicit integrity state.

dual_pp_eligibility

Fail-closed eligibility check kept pure so the negative manifest cell cannot accidentally initialize CUDA state. Slot preparation itself remains PpNRt::prepare_overlap_slots.

dual_pp_mode

dual_pp_mode_resolve

Pure resolution for MEMRA_DUAL_PP, split from the env read so the flip regression tests cannot race parallel test threads on process env.

dual_pp_on

True when the dual-active door is open (Forced or Auto). Read per step so the model-level gate can replay serial and waved arms against one loaded checkpoint.

dual_pp_overlaps

dual_pp_route

Engine-entry routing for the dual-active path, kept pure for the flip regression tests. Forced routes every B>=2 PP-2 call into decode_step_batch_dual even when the placement cannot run it, so the binding refusals stay reachable and loud. Auto routes only the exact re-gated regime and leaves everything else on the serial PP-N walker. dual_pp_eligibility remains behind this as defense in depth.

dual_pp_slot_snapshot

(completed wave pairs, [slot 0 uses, slot 1 uses], rejected same-slot pairs).

dual_pp_timing_dropped

dual_pp_timing_on

CUDA-event timing is a diagnostic-only process door. The scored N=5 block runs without it; the companion box1 diagnostic process enables it and exports cumulative per-wave stage spans through /metrics.

dual_pp_timing_snapshot

(total_nanoseconds, samples) for wave-A stage0/stage1 then wave-B stage0/stage1.

dual_pp_wave_mid

Pure schedule policy shared by the runtime and kernel-check manifest cells. A single row has no second wave and must stay on the serial PP-N walker.

init_model_transport

Bring up the PP transport while model geometry is known but before model weights upload. Door-shut and placement-free loads remain untouched.

layer_engine

M2 increment 2 (weight sharding): the engine that should UPLOAD layer il’s weights (and build its decode mirrors) — the owning stage’s engine when the door is open with device placement and sharding not rolled back; else the primary. il >= n_trunk (MTP/NextN blocks) maps to the last stage. The head (output_norm + lm head) belongs to the last trunk layer’s stage — call with il = n_trunk - 1.

new_cache

Stage-owned cache allocation door: when the ppN door is open AND MEMRA_PP_DEVICES is set (placement plumbing), each layer’s cache is allocated by its OWNING stage’s engine — on one device this is byte-for-byte today’s allocation (gated); cross-device it puts each stage’s KV on that stage’s HBM. Door shut or devices unset: plain Cache::new (zero behavior change). Trailing MTP/NextN layers (beyond the trunk) map to the LAST stage.

new_cache_planned

peer_probe_metrics

peer_probe_startup_policy

Pure startup policy so unit tests and kernel-check pin the entire refusal matrix without mutating process-global environment variables.

pp2_overlap

MEMRA_PP_OVERLAP: alternate the double-buffered boundary slots per step (the pipelining seed). Scheduling structure only, never math. Read per step so gates can A/B in-process.

pp2_overlap_resolve

Pure resolution for MEMRA_PP_OVERLAP, split from the env read for the flip regression tests.

pp2_split

N=2 back-compat view of the door (the gemma4 arm and pp2-gate are 2-stage): Some(cut) iff the door is open with EXACTLY two stages.

pp2_streams_off

MEMRA_PP_STREAMS=0: rollback to the increment-1 same-stream seam (boundary = two plain dtod copies on the ambient compute stream, no per-stage streams/events/devices).

pp_cuts

Returns the stage fence iff the ppN door is open: MEMRA_PP_STAGES=N (N >= 2) with a valid cut list. The fence has N+1 entries: [0, c1, .., cN-1, n_layers]; stage s runs layers [fence[s], fence[s+1]). Reads the environment on every call (gates toggle the door in-process); the cost is a few getenv per decode step, eager-loop noise.

pp_exit_publish

MEMRA_PP_EXIT_PUBLISH — the ppN EXIT-PUBLICATION guard (lane/glm5-accrace 2026-09-01).

pp_host_bounce_active

True when host bounce is the live transport for a sharded cross-device placement. Callers use this to close paths that still peer-read non-boundary state.

pp_host_bounce_on

Broken-peer escape hatch: stage-boundary activations travel through page-locked host memory instead of cudaMemcpyPeerAsync. Default OFF; captured when PpNRt is built.

pp_multi_stream_same_device

True iff the ppN door would put TWO OR MORE stage streams on ONE device (devices unset = all stages on the primary; or an explicit placement with a repeated device). The deferred-readback (pipelined) arm is REFUSED in this regime: the 2026-08-02 x20 soak record — singledev pipelined 13/20 PASS default, 7 failures each diverging at a different step (timing-race signature); MEMRA_PDL=0 went 20/20 on one soak but a second same-config soak on the auto-gated build failed 2/20 (n2) and battery-4 failed n4 — so PDL narrows the window without closing it, and the true root cause (same Engine kernels concurrent on two streams of one device) is NOT fixed by any flag yet. Cross-device pipelined (one stage stream per device) is 23/23 clean post-fix. Refuse loudly rather than return silently-wrong logits. Env-only read (callable pre-runtime).

pp_shard_off

M2 increment 2 rollback seam: MEMRA_PP_SHARD=0 = the M1 bring-up placement (all weights upload through the primary engine; remote stages peer-read). Default ON — under MEMRA_PP_DEVICES each stage’s layer range uploads through its own engine.

pp_sharded_cross_device

True iff the ppN door is open AND the placement spans 2+ DISTINCT devices AND the per-stage sharded loader is on — i.e. some layers’ weights live on a device other than the primary. Any path that walks the WHOLE trunk on one stream in this regime reads those weights over PCIe every step. Env-only read (callable pre-runtime).

pp_wave_diagonal

Cells on one pipeline anti-diagonal, returned as (wave, stage). Cells in a diagonal never share a wave (request/cache state) or a stage (Engine scratch/stream), so they may be driven by scoped host threads without reintroducing the shared-Engine race that quarantined the old deferred PP walker.

pp_wave_eligibility

Fail-closed topology gate for the unqualified PP3/PP4 wavefront. Native peer transport and one physical device per stage are required for the first implementation; host bounce and repeated devices retain the serial PP-N walker.

pp_wave_numeric_eligibility

pp_wave_on

pp_wave_on_value

Strict opt-in for the PP3/PP4 request wavefront. PP2 keeps its independently qualified MEMRA_DUAL_PP default; a new stage count never inherits that default without its own target receipts.

pp_wave_ranges

Split one scheduler tick into at most one wave per stage. Earlier waves carry the remainder so priority order is preserved, every row appears exactly once, and the largest wave is ceil(batch / min(batch, stages)).

pp_wave_route_enabled

One routing predicate shared by decode and prime: requesting the wave door without the double-slot policy is the documented serial rollback, not a late per-request refusal.

pp_wave_snapshot

(completed ticks, entered cells, observed concurrent-cell overlaps).

prime_pipe_on

MEMRA_PRIME_PIPE=0: rollback/A-B seam for the PP-2 PRIME CHUNK PIPELINE (lane/cx-pipeline-prime 2026-08-08). Default ON when the prime stage split is live; setting 0 keeps the serial per-chunk stage walk. Read per prime call so the exactness gate can replay both schedules against one loaded model.

prime_pipe_overlaps

Read the prime-pipeline overlap counter (gate-side).

prime_pp_on

MEMRA_PRIME_PP=0: rollback/A-B seam for the PRIME (chunked prefill) stage split (lane/pp-leverb 2026-08-08). Default ON — with the ppN door open the chunked prime takes its own per-stage range walk exactly as the eager/batched/verify steps do. Setting 0 sends prime back through the unsplit whole-trunk walk. NOTE: unlike batch/dc/graph/spec, prime keeps NO refuse_unsplit_if_remote — its unsplit walk over a sharded placement is the measured 22% amortized peer-read tax (research/pp-prefill-20260807 anatomy: m=4096 amortizes the weight reads), not the decode 28x cliff, and the unsplit walk IS the split-vs-unsplit gate’s reference arm (prime-split-gate), so it must stay callable. Read per call, never memoized (the gate A/Bs both arms in one process).

prime_split_chunks

Read the split-liveness counter (gate-side).

record_runtime_peer_probe_deferral

Publish newly observed copy-count intervals where a runnable peer probe was blocked by live speculative UVA state. The worker coalesces scheduler polls before calling this function.

refuse_unsplit_if_remote

The shared fail-closed guard for EVERY decode path that has no pp stage split. Returns Err iff pp_sharded_cross_device() — i.e. the caller would walk the whole trunk on one stream while some layers’ weights live on another device, peer-reading them every step. path names the refusing function so the operator knows which loop they hit; alt names the working alternative for that loop.

restore_cache_checkpoint

Restore a cache checkpoint through each layer’s owning engine.

service_runtime_peer_probe

Service a due runtime peer probe without constructing a PP runtime on door-shut placements. Must be called by the CUDA owner thread at a scheduling boundary.

spec_pp_on

MEMRA_SPEC_PP=0: rollback/A-B seam for the SPEC VERIFY stage split (pp2-spec 2026-08-06). Default ON — with the ppN door open the verify forward (decode_step_t_core_ppn) takes its own stage split exactly as the eager and batched steps do. Setting 0 sends verify back through the unsplit trunk walk, which under a sharded cross-device placement is then caught by refuse_unsplit_if_remote (the 28x peer-read regime) rather than running silently. Exists so the bit-identity gate can A/B split vs unsplit IN ONE PROCESS against the same loaded weights — read per verify call, never memoized, for that reason.

stage_of

The stage that owns layer il under fence (see pp_cuts).

step35_prime_batch_splits

step35_prime_batches

sync_stages_after_load

M2 increment 2 LOAD BARRIER: weight uploads and decode-mirror builds enqueue on the loading engines’ WORKER streams; the first consumer launches on a DIFFERENT stream with no load->decode event — the door-off reference walk on the primary worker stream (sharded load: remote builds still in flight), or a fresh per-stage stream. The 2026-08-02 gate finds (n2-dev01 step-0 168k-logit graze; split5 ref=0.0 head — a half-built rp4 mirror — poisoning step-0 KV and every later step): one context-wide synchronize per stage at load end kills the class. No-op when the door is shut at load (single-stream load+decode is ordered by the stream itself).

warn_unwired_once

One-time notice when the door is set but the executing path has no pp arm (M2 wires the generic eager decode at any N and the gemma4 eager arm at N=2).

Type Aliases

Pp2Rt

M1 name kept alive for external callers (pp-transport-smoke, receipts, docs).