Enum OpKind 

#[non_exhaustive]
pub enum OpKind {
    MatMul,
    AddElementwise,
    SubElementwise,
    MulElementwise,
    DivElementwise,
    ReluElementwise,
    NegElementwise,
    SqrElementwise,
    SqrtElementwise,
    RecipElementwise,
    AbsElementwise,
    TanhElementwise,
    ExpElementwise,
    LogElementwise,
    SinElementwise,
    CosElementwise,
    SigmoidElementwise,
    SiluElementwise,
    GeluTanhElementwise,
    StepElementwise,
    SumReduce,
    MaxReduce,
    MinReduce,
    MeanReduce,
    Cast,
    Conv2D,
    ConvTranspose2D,
    ReduceSumTo,
    ReduceMaxTo,
    FusedLinear,
    FlashAttn,
    FlashAttnBackwardQ,
    FlashAttnBackwardK,
    FlashAttnBackwardV,
    PagedAttn,
    Affine,
    ClampElementwise,
    PowIElementwise,
    PowIElementwiseBackward,
    MaximumElementwise,
    MinimumElementwise,
    EqualElementwise,
    NotEqualElementwise,
    LessElementwise,
    LessEqualElementwise,
    GreaterElementwise,
    GreaterEqualElementwise,
    Where,
    FloorElementwise,
    CeilElementwise,
    RoundElementwise,
    SignElementwise,
    ErfElementwise,
    GeluElementwise,
    PowElementwise,
    RsqrtElementwise,
    RemElementwise,
    Flip,
    Roll,
    CumSum,
    Pad,
    Triu,
    Tril,
    LogSoftmaxLastDim,
    LogSoftmaxLastDimBackward,
    MaskedFill,
    PadBackward,
    Concat,
    SoftmaxLastDim,
    SoftmaxLastDimBackward,
    RmsNormLastDim,
    RmsNormLastDimBackward,
    LayerNormLastDim,
    LayerNormLastDimBackward,
    ReduceMaxToBackward,
    IndexSelect,
    Gather,
    Rope,
    IndexAdd,
    ScatterAdd,
    ArgMaxDim,
    ArgMinDim,
    QMatMul,
    WriteSlice,
    WriteSliceRotating,
    WriteSliceDoff,
    Copy,
    ReluInplace,
    SiluInplace,
    GeluTanhInplace,
    TanhInplace,
    SigmoidInplace,
    NegInplace,
    AbsInplace,
    SqrInplace,
    SqrtInplace,
    RsqrtInplace,
    RecipInplace,
    ExpInplace,
    LogInplace,
    SinInplace,
    CosInplace,
    SignInplace,
    FloorInplace,
    CeilInplace,
    RoundInplace,
    ErfInplace,
    GeluInplace,
    ClampInplace,
    PowIInplace,
    InplaceAffine,
    FusedSoftmaxCrossEntropy,
    CausalConv1d,
    SelectiveScan,
    SsdChunkScan,
    Nf4Matmul,
    RuntimeFused,
    NonZeroIndices,
}

Op kinds the Judge profiles. Adding a variant + a Judge match arm extends the profile matrix; existing reports parse forward thanks to #[non_exhaustive].

Phase 7.5 storage unification — Phase C grows this enum one op family per migration step. Each variant is the dispatch key that pairs with crate::DType and crate::probe::BackendId to select a kernel. The naming convention is <Op><Family> (e.g. ReluElementwise, MulElementwise); families currently in use:

• elementwise binary: Add/Sub/Mul/Div + Elementwise
• elementwise unary: Relu/Neg/Sqr/Sqrt/Recip/Abs/Tanh/...
  • Elementwise
• dense linear algebra: MatMul

Variants (Non-exhaustive)

Non-exhaustive enums could have additional variants added in future. Therefore, when matching against variants of non-exhaustive enums, an extra wildcard arm must be added to account for any future variants.

MatMul

Dense matrix multiply, [M, K] @ [K, N] → [M, N].

AddElementwise

Elementwise addition of two equally-shaped tensors.

SubElementwise

Elementwise subtraction (a - b).

MulElementwise

Elementwise multiplication.

DivElementwise

Elementwise division (a / b).

ReluElementwise

Elementwise rectified linear unit (max(0, x)).

NegElementwise

Elementwise negation (-x).

SqrElementwise

Elementwise square (x * x).

SqrtElementwise

Elementwise square root.

RecipElementwise

Elementwise reciprocal (1 / x).

AbsElementwise

Elementwise absolute value.

TanhElementwise

Elementwise hyperbolic tangent.

ExpElementwise

Elementwise exponential (e^x).

LogElementwise

Elementwise natural logarithm (ln(x)).

SinElementwise

Elementwise sine.

CosElementwise

Elementwise cosine.

SigmoidElementwise

Elementwise logistic sigmoid (1 / (1 + e^-x)).

SiluElementwise

SiLU / Swish activation (x * sigmoid(x)).

GeluTanhElementwise

GELU activation, tanh approximation (0.5*x*(1 + tanh(√(2/π) * (x + 0.044715*x³)))).

StepElementwise

Heaviside step function (1 where x > 0, 0 otherwise) — the derivative of OpKind::ReluElementwise.

SumReduce

Sum-reduce one or more dimensions of the input. The reduced dims and the input shape live in OpParams::Reduce; the output is the input with those dims dropped (or rank-0 when every dim is reduced).

MaxReduce

Max-reduce — same shape contract as OpKind::SumReduce.

MinReduce

Min-reduce — same shape contract as OpKind::SumReduce.

MeanReduce

Arithmetic-mean reduce — same shape contract as OpKind::SumReduce; divides the sum by the product of reduced dim sizes.

Cast

Convert input bytes from one dtype to another. The target dtype lives on the output Storage’s dtype field (and on the Node’s dtype); the source dtype is read from the input Storage at runtime by the wrapper, which dispatches to the right typed conversion kernel internally. Binding-table lookup is keyed on the target dtype (= the Node’s dtype).

Conv2D

2D convolution forward pass. Shape + geometry flow through OpParams::Conv2D. Two-input case: (x, weight); three-input case adds bias. Output shape [N, Cout, Hout, Wout].

ConvTranspose2D

2D transposed convolution. Shape + geometry flow through OpParams::ConvTranspose2D. Inputs and bias mirror Conv2D, but weight has transposed channel order ([Cin, Cout/groups, Kh, Kw]) and the variant carries output_padding.

ReduceSumTo

Sum-reduce a tensor to a smaller broadcast-compatible shape. The output rank may be lower than the input’s; in that case the output shape is implicitly left-padded with 1s for axis alignment. Carried by OpParams::ReduceSumTo with both the input and target shapes.

ReduceMaxTo

Max-reduce a tensor to a smaller broadcast-compatible shape — the max-symmetric counterpart of ReduceSumTo. Same axis alignment rules; per-axis reduction is max instead of +. Carried by OpParams::ReduceMaxTo with both shapes.

FusedLinear

Fused matmul + bias-add. Inputs [a, b, bias], output [..., M, N] = a @ b + bias[None..., :]. Reuses OpParams::Matmul for shape (kernel inits accumulator with the bias element instead of zero).

FlashAttn

Multi-head scaled-dot-product attention (the math definition, not a tiled FlashAttention-2 kernel — naive on CPU). Inputs [q, k, v, optional alibi_slopes]. Geometry, softmax_scale, causal, window, softcap all flow through OpParams::FlashAttn.

FlashAttnBackwardQ

Backward of OpKind::FlashAttn: produces dQ from (q, k, v, do, [alibi]). Reuses OpParams::FlashAttn (same shape/geometry/causal flags as the forward — the recompute pass needs every forward parameter). Output shape == q shape. The dK and dV gradients are emitted as separate OpKind::FlashAttnBackwardK / OpKind::FlashAttnBackwardV nodes against the same inputs; CPU backends recompute the softmax state independently per call. A 3-output fused variant would share the recompute but needs multi-output infrastructure that doesn’t exist yet.

FlashAttnBackwardK

Backward of OpKind::FlashAttn: produces dK. See OpKind::FlashAttnBackwardQ. Output shape == k shape.

FlashAttnBackwardV

Backward of OpKind::FlashAttn: produces dV. See OpKind::FlashAttnBackwardQ. Output shape == v shape.

PagedAttn

Paged-cache scaled-dot-product attention. Inputs [q, k_cache, v_cache, block_table, context_lens, optional alibi_slopes]. Geometry + scale + softcap flow through OpParams::PagedAttn.

Affine

Affine transformation y = mul * x + add with scalar coefficients. Op::AddScalar(c) maps as mul=1, add=c; Op::MulScalar(c) maps as mul=c, add=0.

ClampElementwise

Element-wise clamp: y = clamp(x, min, max).

PowIElementwise

Element-wise integer power: y = x.powi(exp).

PowIElementwiseBackward

Backward of OpKind::PowIElementwise: (x, upstream) → grad_x = exp · x^(exp-1) · upstream. Two inputs; carries the same exp: i32 in OpParams::PowI as the forward. Single-launch alternative to the autograd primitive decomposition (PowI(exp-1) → MulScalar → Mul, 3 nodes). Per-dtype.

MaximumElementwise

Element-wise tensor maximum: y[i] = max(lhs[i], rhs[i]).

MinimumElementwise

Element-wise tensor minimum: y[i] = min(lhs[i], rhs[i]).

EqualElementwise

Element-wise equality a == b. Both inputs share dtype T; output is a U8 mask (1 where equal, 0 otherwise). The binding-table dtype list is keyed [T, T, U8], mirroring the ArgMax convention of carrying the output dtype explicitly when it differs from the inputs.

NotEqualElementwise

Element-wise inequality a != b. Same shape contract as OpKind::EqualElementwise; output U8 mask (1 where unequal, 0 otherwise). NaN follows IEEE-754: NaN != NaN is true, so ne returns 1 on NaN-vs-NaN positions.

LessElementwise

Element-wise strictly-less a < b. Same shape contract as OpKind::EqualElementwise; output U8 mask. NaN-on-either-side is always false (IEEE-754 unordered comparison).

LessEqualElementwise

Element-wise less-or-equal a <= b. Same shape contract. NaN comparisons are unordered → 0.

GreaterElementwise

Element-wise strictly-greater a > b. Same shape contract. NaN comparisons are unordered → 0.

GreaterEqualElementwise

Element-wise greater-or-equal a >= b. Same shape contract. NaN comparisons are unordered → 0.

Where

Ternary select: out[i] = if cond[i] != 0 { a[i] } else { b[i] }. Inputs (cond, a, b) — cond is U8, a and b share dtype T and shape with cond. Output is T shape cond.shape(). Binding-table dtype list [U8, T, T, T].

FloorElementwise

Element-wise floor (⌊x⌋). Same dtype as input. Backward is the zero distribution almost everywhere; gradient through rounding ops is dropped silently.

CeilElementwise

Element-wise ceiling (⌈x⌉). Same dtype as input. Backward drops gradient (mirrors Floor).

RoundElementwise

Element-wise round-to-nearest with banker’s rounding (round-half-to-even, IEEE 754 roundeven). Backward drops gradient (mirrors Floor/Ceil).

SignElementwise

Element-wise sign (-1 if negative, 0 if zero, 1 if positive). Same dtype as input. Backward drops gradient (zero almost everywhere; the subgradient at 0 is taken as 0).

ErfElementwise

Element-wise Gauss error function (erf(x)). Same dtype as input. Backward: d/dx erf(x) = (2/√π) * exp(-x²) — expressed via Sqr + Neg + Exp + MulScalar.

GeluElementwise

Element-wise GELU activation, exact erf formulation: 0.5 * x * (1 + erf(x/√2)). Distinct from OpKind::GeluTanhElementwise (tanh approximation). Same dtype as input. Backward decomposes into the standard-normal CDF + x * φ(x) (PDF) chain via existing primitives.

PowElementwise

Element-wise binary power: out[i] = pow(a[i], b[i]). Both inputs share dtype T and shape; output is T with the same shape. Distinct from OpKind::PowIElementwise (scalar i32 exponent). NaN follows IEEE-754 (e.g. pow(-2, 0.5) = NaN).

RsqrtElementwise

Element-wise reciprocal square root: out[i] = 1 / sqrt(x[i]). Single op (vs Sqrt + Recip — combining loses precision and doubles kernel launches). Critical for RMSNorm (x / sqrt(mean(x²)+eps) → x * rsqrt(mean(x²)+eps)).

RemElementwise

Element-wise remainder, PyTorch convention: out[i] = a[i] - floor(a[i] / b[i]) * b[i]. The sign of the result matches the sign of the divisor (differs from C99 fmod which has the sign of the dividend; f32::% is also fmod-style). Same dtype + shape on both inputs.

Flip

Reverse the order of elements along one dim. Dtype-agnostic at the byte level. Materializing op (Layout strides are unsigned; the negative-stride view path requires a wider stride representation that’s a separate scope).

Roll

Cyclic shift along one dim by a signed shift. Wraps (e.g. roll([0,1,2], shift=1) = [2,0,1]). Materializing op, dtype-agnostic.

CumSum

Running cumulative sum along one dim: out[..., i, ...] = sum(in[..., 0..=i, ...]). Same shape as input. Per-dtype kernel (sum needs typed addition).

Pad

Pad along one dim with before/after extra slots. Modes: Constant (fill with a value), Reflect (mirror edges), Replicate (repeat edges). Only Constant is implemented in the v1 cut; Reflect/Replicate fall through to a clean error.

Triu

Upper-triangular mask along the last two dims. Materializing, dtype-agnostic at the byte level (output is x or zero per position; just selects bytes from src or the zero-init buffer).

Tril

Lower-triangular mask along the last two dims (mirror of OpKind::Triu).

LogSoftmaxLastDim

Numerically-stable log-softmax along the last dim. Per-dtype (uses log/exp). Output shape == input shape.

LogSoftmaxLastDimBackward

Backward of OpKind::LogSoftmaxLastDim: takes (forward_output, upstream) and produces the input gradient. Per-dtype.

MaskedFill

MaskedFill: fill positions where mask is nonzero with a scalar. Inputs (x, mask) where mask is U8. Per-dtype on x; output shape == x shape.

PadBackward

Backward helper for Pad. Per-dtype since accumulation is typed addition. Routes all 3 modes uniformly — the kernel switches on mode_tag.

Concat

Concatenate N inputs along one dim. Inputs must agree on every dim except the concat dim; output’s concat-dim size is the sum of inputs’ concat-dim sizes.

SoftmaxLastDim

Softmax along the last dim, numerically stable (subtract per-row max, exp, divide by sum).

SoftmaxLastDimBackward

Backward of OpKind::SoftmaxLastDim: (y, g) → y · (g - sum(y · g, last)). Per-dtype; output shape == y shape.

RmsNormLastDim

RMS normalization along the last dim, no affine params: y = x / sqrt(mean(x², last) + eps).

RmsNormLastDimBackward

Backward of OpKind::RmsNormLastDim: (x, g_y) → grad_x per the closed-form formula in fuel-reference-backend::ops::rms_norm_last_dim_backward. Per-dtype + eps; output shape == x shape.

LayerNormLastDim

Layer normalization along the last dim, no affine params: y = (x - mean(x)) / sqrt(var(x) + eps).

LayerNormLastDimBackward

Backward of OpKind::LayerNormLastDim: (x, g) → grad_x per the canonical formula. Per-dtype + eps; output shape == x shape.

ReduceMaxToBackward

Backward of OpKind::ReduceMaxTo: (x, upstream) → grad_x of x’s shape. Routes upstream to argmax positions; ties split equally. Per-dtype; carries shape pair via OpParams::ReduceMaxToBackward.

IndexSelect

Pick slices from a source tensor along dim using a rank-1 U32 index tensor. Output’s dim size = number of indices.

Gather

N-dimensional gather along dim. Source and indices have the same rank; output shape equals indices.shape(). For each output position, source is read at the same multi-index except dim’s coord is read from the indices tensor.

Rope

Fused rotary position embedding. Inputs (x, cos, sin); cos/sin broadcast across leading dims. Used by every modern transformer’s attention layer.

IndexAdd

Add src values into a copy of base at positions given by a rank-1 U32 indices tensor along dim. Inputs: (base, indices, src). Output shape == base shape.

ScatterAdd

N-dimensional scatter-add: the functional inverse of Gather. Inputs (base, indices, src); indices and src share the same shape; for each position p, the source value is added into base at the multi-index where dim’s coord is indices[p].

ArgMaxDim

Argmax along one dim — produces a U32 index tensor with dim removed from the output shape.

ArgMinDim

Argmin along one dim — same shape contract as OpKind::ArgMaxDim.

QMatMul

Quantized matmul: C = A @ dequant(W_Q). Activations are f32 (or eventually bf16); weights are a U32-typed byte stream of quantized blocks. The quant format is carried in OpParams::QMatMul.

WriteSlice

In-place rectangular scatter write: copies source’s bytes into a slab of destination defined per-axis by ranges. Backs Phase E.3.2’s persistent KV-cache writes (InferenceContext + KvCache). See Op::WriteSlice for the IR contract and OpParams::WriteSlice for the kernel-side params.

WriteSliceRotating

In-place ring-buffer scatter write: copies source’s bytes into destination at a slab that wraps modulo modulus on the rotating axis. The write position comes from a dynamic input (rank-0 U32). Backs sliding-window KV caches (Mistral / Phi-3 sliding-window / sliding-window Qwen). See Op::WriteSliceRotating for the IR contract.

WriteSliceDoff

In-place scatter write whose start on ONE axis is read from a device-resident i64 scalar input (rank-0) at kernel launch, not host-baked into the range — the CUDA-graph-capturable KV-cache append. Like WriteSlice but the ranges[axis].0 placeholder is overridden by *offset device- side; no modulo wrap. Backs DecodeSession / CapturedRun. See Op::WriteSliceDoff for the IR contract.

Copy

Cross-device copy: produce a fresh tensor on the target device, copying bytes from the input’s residency. Backs Op::Copy for the bridge-retirement trajectory’s Phase 2 (D2H through the binding table).

Binding-table key shape [T, T] (input dtype, output dtype — always equal). The BackendId axis encodes the source backend (where the kernel runs — the source owns its own download path); the executor allocates the output Storage on the Op::Copy { target } location field via a dedicated WorkItemKind::Copy arm.

ReluInplace

In-place OpKind::ReluElementwise — x = max(0, x).

SiluInplace

In-place OpKind::SiluElementwise — x = x · sigmoid(x).

GeluTanhInplace

In-place OpKind::GeluTanhElementwise (tanh approximation) — x = 0.5 · x · (1 + tanh(√(2/π) · (x + 0.044715·x³))).

TanhInplace

In-place OpKind::TanhElementwise — x = tanh(x).

SigmoidInplace

In-place OpKind::SigmoidElementwise — x = 1 / (1 + exp(-x)).

NegInplace

In-place OpKind::NegElementwise — x = -x.

AbsInplace

In-place OpKind::AbsElementwise — x = |x|.

SqrInplace

In-place OpKind::SqrElementwise — x = x².

SqrtInplace

In-place OpKind::SqrtElementwise — x = √x.

RsqrtInplace

In-place OpKind::RsqrtElementwise — x = 1/√x.

RecipInplace

In-place OpKind::RecipElementwise — x = 1/x.

ExpInplace

In-place OpKind::ExpElementwise — x = exp(x).

LogInplace

In-place OpKind::LogElementwise — x = ln(x).

SinInplace

In-place OpKind::SinElementwise — x = sin(x).

CosInplace

In-place OpKind::CosElementwise — x = cos(x).

SignInplace

In-place OpKind::SignElementwise — x = sign(x).

FloorInplace

In-place OpKind::FloorElementwise — x = ⌊x⌋.

CeilInplace

In-place OpKind::CeilElementwise — x = ⌈x⌉.

RoundInplace

In-place OpKind::RoundElementwise — x = round(x).

ErfInplace

In-place OpKind::ErfElementwise — x = erf(x).

GeluInplace

In-place OpKind::GeluElementwise — exact-GeLU x = 0.5 · x · (1 + erf(x/√2)).

ClampInplace

In-place OpKind::ClampElementwise — x = clamp(x, min, max). Scalar (min, max) flow through OpParams::Clamp.

PowIInplace

In-place OpKind::PowIElementwise — x = x.powi(exp). Scalar exp flows through OpParams::PowI.

InplaceAffine

In-place OpKind::Affine — x = mul · x + add. The (mul, add) coefficients flow through OpParams::Affine; the kernel reads + writes the same buffer. Single-input, single-output. Backs FusedOps::INPLACE_AFFINE.

FusedSoftmaxCrossEntropy

Fused softmax + negative log-likelihood with integer class targets — the standard PyTorch / Liger-Kernel training loss. Inputs [logits, targets] where logits: [n_rows, vocab] (F32, flattened from rank-N) and targets: [n_rows] (I64). Output F32 — scalar for Mean/Sum reductions, [n_rows] for None. Geometry + reduction + ignore_index flow through OpParams::FusedSoftmaxCrossEntropy. Backs FusedOps::FUSED_SOFTMAX_CROSS_ENTROPY.

CausalConv1d

Depthwise 1-D causal convolution + bias + optional fused SiLU. Inputs [x, weight, bias] where x: [batch, channels, seq + kernel - 1] (left-pre-padded by caller), weight: [channels, 1, kernel], bias: [channels]. Output [batch, channels, seq]. Geometry + use_silu flow through OpParams::CausalConv1d. Backs FusedOps::CAUSAL_CONV1D — the Mamba-1 / Mamba-2 prefill convolution fusion.

SelectiveScan

Mamba-1’s selective state-space scan (forward). Five inputs [u, delta, a, b, c] — see FusedOps::SELECTIVE_SCAN for the full shape contract. Output y: [batch, seqlen, dim]. Geometry + delta_softplus flow through OpParams::SelectiveScan.

SsdChunkScan

Mamba-2’s State-Space Duality chunked scan (forward). Five inputs [x, dt, a, b, c] — see FusedOps::SSD_CHUNK_SCAN for the full shape contract. Output y: [batch, seqlen, heads, head_dim]. Geometry + chunk_size flow through OpParams::SsdChunkScan.

Nf4Matmul

bitsandbytes-style 4-bit NormalFloat quantized matmul. Three inputs [activations, w_packed, absmax] — see FusedOps::NF4_MATMUL for the full shape contract. Output [..., M, N] matches the activations’ dtype. Geometry + block_size flow through OpParams::Nf4Matmul.

RuntimeFused

A runtime-registered (Tier-2, JIT-synthesized / import-time) fused op. Carries no static kernel family: the concrete op is named by the runtime FusedOpId on the graph node, and its kernel resolves from the runtime sidecar (fuel_dispatch::runtime_fused_kernels), never the OpKind-keyed binding table. Exists so a CompiledNode for such an op has an honest op tag; no binding-table registration may use it.

NonZeroIndices

Data-determined nonzero-index extraction — the keystone primitive for data-dependent dynamic shapes. One input x; one bundled output [indices [capacity] U32 ; count [1] U32]. Geometry + count_sym flow through OpParams::NonZeroIndices. See Op::NonZeroIndices.

Implementations

impl OpKind

pub fn as_str(self) -> &'static str

Trait Implementations

impl Clone for OpKind

fn clone(&self) -> OpKind

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for OpKind

impl Debug for OpKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for OpKind

fn deserialize<__D>( __deserializer: __D, ) -> Result<OpKind, <__D as Deserializer<'de>>::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Display for OpKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Eq for OpKind

impl From<OpKind> for BindingKey

fn from(op: OpKind) -> Self

Converts to this type from the input type.

impl Hash for OpKind

fn hash<__H>(&self, state: &mut __H)

where __H: Hasher,

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for OpKind

fn eq(&self, other: &OpKind) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl Serialize for OpKind

fn serialize<__S>( &self, __serializer: __S, ) -> Result<<__S as Serializer>::Ok, <__S as Serializer>::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for OpKind