pub enum Op {
Show 119 variants
Const,
Iota {
len: usize,
},
Add,
Sub,
Mul,
Div,
Neg,
Sqr,
Sqrt,
Exp,
Log,
Sin,
Cos,
Tanh,
Sigmoid,
Silu,
GeluTanh,
Relu,
Step,
Recip,
Abs,
ReluInplace,
SiluInplace,
GeluTanhInplace,
TanhInplace,
SigmoidInplace,
NegInplace,
AbsInplace,
SqrInplace,
SqrtInplace,
RsqrtInplace,
RecipInplace,
ExpInplace,
LogInplace,
SinInplace,
CosInplace,
SignInplace,
FloorInplace,
CeilInplace,
RoundInplace,
ErfInplace,
GeluInplace,
ClampInplace {
min: f64,
max: f64,
},
PowIInplace(i32),
Equal,
Ne,
Lt,
Le,
Gt,
Ge,
Floor,
Ceil,
Round,
Sign,
Erf,
Gelu,
Pow,
Rsqrt,
Rem,
Flip {
dim: usize,
},
Roll {
dim: usize,
shift: i64,
},
CumSum {
dim: usize,
},
Triu {
diagonal: i64,
},
Tril {
diagonal: i64,
},
LogSoftmaxLastDim,
MaskedFill {
value: Scalar,
},
PadBackward {
in_shape: Shape,
padding: Vec<(usize, usize)>,
mode: PadMode,
},
Pad {
padding: Vec<(usize, usize)>,
mode: PadMode,
value: f64,
},
Where,
MatMul,
Transpose,
Permute(Vec<usize>),
Cast(DType),
BroadcastTo(Shape),
Reshape(Shape),
Contiguize,
Unsqueeze {
dim: usize,
},
Squeeze {
dim: usize,
},
ReduceSumTo(Shape),
ReduceMaxTo(Shape),
SumAll,
MaxAll,
MinAll,
MeanAll,
SumDim(usize),
MaxDim(usize),
MinDim(usize),
MeanDim(usize),
LogSoftmaxLastDimBackward,
ArgMaxDim(usize),
ArgMinDim(usize),
Concat {
dim: usize,
},
Slice {
dim: usize,
start: usize,
len: usize,
},
AddScalar(f64),
MulScalar(f64),
PowI(i32),
Clamp {
min: f64,
max: f64,
},
Maximum,
Minimum,
IndexSelect {
dim: usize,
},
Gather {
dim: usize,
},
IndexAdd {
dim: usize,
},
ScatterAdd {
dim: usize,
},
NonZeroIndices {
count_sym: SymId,
},
Copy {
target: DeviceLocation,
},
Release,
Move {
target: DeviceLocation,
},
WriteSlice {
ranges: Vec<(usize, usize)>,
dyn_offset: Option<(usize, DynScalar)>,
},
WriteSliceRotating {
axis: usize,
modulus: usize,
ranges: Vec<(usize, usize)>,
},
WriteSliceDoff {
axis: usize,
ranges: Vec<(usize, usize)>,
},
Alloc {
target: DeviceLocation,
},
ZeroFill,
Fused(FusedOpId, FusedOpParams),
View {
slot: u32,
},
ViewOwned {
slot: u32,
},
ScatterIntoSlot {
slot: u32,
},
Branch {
reconverge_at: NodeId,
},
Scan {
n_carries: usize,
n_xs: usize,
bound: usize,
emit: ScanEmit,
early_exit: Option<ScanPredicate>,
},
ScanPlaceholder {
role: ScanRole,
index: usize,
},
}Expand description
The closed enum of operations a graph node can represent.
This is the API contract between the graph layer and every backend. Adding a new op means (1) adding a variant here, (2) teaching every backend how to execute it, and (3) teaching the reference backend how to compute its textbook answer.
Variants§
Const
A concrete constant tensor — a leaf with no input nodes. The
realized bytes live in the graph’s storage_map slot for this
node, populated at construction time. The executor’s slot-first
dispatch returns the slot’s Arc<RwLock<Storage>> directly.
(Phase 7.5 work item G2 retired the host-side ConstData
payload — the slot is the sole source of bytes.)
Iota
Position-index generator — a leaf (no inputs) producing the 1-D
F32 sequence [0.0, 1.0, …, (len-1) as f32] of shape [len].
The integer-valued positions are exact in f32 for any realistic
sequence length (< 2^24). Reshaped + broadcast, it builds the
row/column indices that express causal / sliding-window masks and
alibi position biases (e.g. in FlashAttn’s decomposition), so those
ops can lower to primitives instead of being opaque. Non-differentiable
(a constant generator; backward drops gradient like Op::Const).
Add
Element-wise addition.
Sub
Element-wise subtraction (a - b).
Mul
Element-wise multiplication.
Div
Element-wise division (a / b).
Neg
Element-wise negation (-x).
Sqr
Element-wise square (x * x).
Sqrt
Element-wise square root.
Exp
Element-wise exponential (e^x).
Log
Element-wise natural logarithm.
Sin
Element-wise sine.
Cos
Element-wise cosine.
Tanh
Element-wise hyperbolic tangent.
Sigmoid
Element-wise logistic sigmoid.
Silu
SiLU activation (x · sigmoid(x)), also called Swish.
GeluTanh
GELU activation (tanh approximation).
Relu
Rectified linear unit (max(0, x)), element-wise.
Step
Heaviside step function: 1 where x > 0, 0 otherwise. Serves
as the derivative of Relu and as a building block for other
comparison-based gradients.
Recip
Element-wise reciprocal (1 / x). On x = 0 the IEEE-754 result
is ±inf; the kernel matches the obvious 1.0 / x expression
rather than treating the input as an error.
Abs
Element-wise absolute value (|x|).
ReluInplace
In-place Relu: x = max(0, x). Same semantics as Op::Relu,
mutates input 0. Not yet wired through autograd.
SiluInplace
In-place Silu: x = x * sigmoid(x). Same semantics as
Op::Silu, mutates input 0. Not yet wired through autograd.
GeluTanhInplace
In-place GeluTanh (tanh approximation): mutates input 0 with the
same formula as Op::GeluTanh. Its backward is the same as Op::GeluTanh’s.
TanhInplace
In-place Tanh: mutates input 0. Same semantics as Op::Tanh.
Not yet wired through autograd.
SigmoidInplace
In-place Sigmoid: mutates input 0. Same semantics as
Op::Sigmoid. Not yet wired through autograd.
NegInplace
In-place Neg: x = -x. Same semantics as Op::Neg.
AbsInplace
In-place Abs: x = |x|. Same semantics as Op::Abs.
SqrInplace
In-place Sqr: x = x². Same semantics as Op::Sqr.
SqrtInplace
In-place Sqrt: x = √x. Same semantics as Op::Sqrt.
RsqrtInplace
In-place Rsqrt: x = 1/√x. Same semantics as Op::Rsqrt.
RecipInplace
In-place Recip: x = 1/x. Same semantics as Op::Recip.
ExpInplace
In-place Exp: x = exp(x). Same semantics as Op::Exp.
LogInplace
In-place Log: x = ln(x). Same semantics as Op::Log.
SinInplace
In-place Sin: x = sin(x). Same semantics as Op::Sin.
CosInplace
In-place Cos: x = cos(x). Same semantics as Op::Cos.
SignInplace
In-place Sign: x = sign(x). Same semantics as Op::Sign.
Backward drops gradient (mirrors Op::Sign).
FloorInplace
In-place Floor: x = ⌊x⌋. Same semantics as Op::Floor.
Backward drops gradient (mirrors Op::Floor).
CeilInplace
In-place Ceil: x = ⌈x⌉. Same semantics as Op::Ceil.
Backward drops gradient (mirrors Op::Ceil).
RoundInplace
In-place Round: x = round(x). Same semantics as Op::Round.
Backward drops gradient (mirrors Op::Round).
ErfInplace
In-place Erf: x = erf(x). Same semantics as Op::Erf.
GeluInplace
In-place exact-GeLU: x = 0.5 · x · (1 + erf(x/√2)). Same
semantics as Op::Gelu.
ClampInplace
In-place clamp: x = clamp(x, min, max). Same semantics as
Op::Clamp. Backward gates the upstream by (x ≥ min) ∧ (x ≤ max) (mirrors Op::Clamp).
PowIInplace(i32)
In-place integer-power: x = x.powi(exp). Same semantics as
Op::PowI. Backward emits exp · upstream · x.powi(exp-1)
(mirrors Op::PowI).
Equal
Element-wise equality (a == b) producing a Bool mask: 1
where the inputs are equal, 0 otherwise. Both operands must
share dtype and shape. NaN follows IEEE-754 (NaN == NaN is
false). Output dtype is always Bool, regardless of input dtype
— the binding-table key is [T, T, Bool].
Non-differentiable: backward returns None for both inputs
(registered via crate::grad::NoGradientBinaryRule).
Ne
Element-wise inequality (a != b) producing a Bool mask: 1
where unequal, 0 otherwise. Same shape/dtype contract as
Op::Equal. NaN follows IEEE-754 (NaN != NaN is true →
1). Non-differentiable.
Lt
Element-wise strictly-less (a < b) producing a Bool mask: 1
where lhs is strictly less than rhs, 0 otherwise. Same
shape/dtype contract as Op::Equal. NaN-on-either-side is
always 0 (IEEE-754 unordered comparison). Non-differentiable.
Le
Element-wise less-or-equal (a <= b) producing a Bool mask.
Same shape/dtype contract as Op::Equal. NaN-on-either-side
is always 0. Non-differentiable.
Gt
Element-wise strictly-greater (a > b) producing a Bool mask.
Same shape/dtype contract as Op::Equal. NaN-on-either-side
is always 0. Non-differentiable.
Ge
Element-wise greater-or-equal (a >= b) producing a Bool mask.
Same shape/dtype contract as Op::Equal. NaN-on-either-side
is always 0. Non-differentiable. Final variant of the
comparison family (Equal, Ne, Lt, Le, Gt, Ge).
Floor
Element-wise floor (⌊x⌋). Same dtype as input. Backward is
the zero distribution almost everywhere; gradient is dropped
silently (treated like Op::Step’s no-grad backward).
Ceil
Element-wise ceiling (⌈x⌉). Same dtype as input. Backward
drops gradient (mirrors Op::Floor).
Round
Element-wise round-to-nearest with banker’s rounding
(round-half-to-even, IEEE 754 roundeven). 0.5 → 0, 1.5 → 2,
2.5 → 2 (NOT 3), 3.5 → 4. Matches NumPy/PyTorch defaults and
differs from C99 round() (half-away-from-zero) at exact ties.
Backward drops gradient (mirrors Op::Floor).
Sign
Element-wise sign (-1 / 0 / 1). sign(0) = 0 by
subgradient convention (same convention Op::Abs’s backward
uses). Same dtype as input. Backward drops gradient (zero
almost everywhere).
Erf
Element-wise Gauss error function (erf(x)). Same dtype as
input. Differentiable: d/dx erf(x) = (2/√π) * exp(-x²).
Backward emitted via existing primitives (Sqr → Neg → Exp →
MulScalar(2/√π) → Mul(upstream, .)).
Gelu
Element-wise GELU activation, exact erf formulation:
0.5 * x * (1 + erf(x/√2)). Distinct from Op::GeluTanh
(tanh approximation, faster but slightly less accurate).
Differentiable; backward decomposes into the standard-normal
CDF + x * φ(x) (PDF) chain via primitives.
Pow
Element-wise binary power: out = pow(a, b) with real b.
Both inputs share dtype T and shape. Distinct from
Op::PowI (scalar i32 exponent). Backward:
d/da pow(a,b) = b * pow(a, b-1)
d/db pow(a,b) = pow(a,b) * ln(a)
NaN follows IEEE-754 (pow(-2, 0.5) = NaN); ln of negative a
in the d/db term yields NaN at those positions.
Rsqrt
Element-wise reciprocal square root: out = 1 / sqrt(x).
Same dtype as input. Distinct from Sqrt followed by Recip
(one op instead of two — saves a kernel launch and matches the
RMSNorm shape x * rsqrt(mean(x²)+eps)). Backward:
d/dx (x^(-1/2)) = -0.5 * x^(-3/2) = -0.5 * y³
where y = 1/sqrt(x) is the forward output (so the chain
reuses the forward node and never touches x directly —
safe near x=0).
Rem
Element-wise remainder, PyTorch convention:
out = a - floor(a / b) * b. The result has the sign of the
divisor (matches torch.remainder). Distinct from C99 fmod
(sign of dividend) and f32::rem_euclid (always non-negative).
Backward:
d/da = 1
d/db = -floor(a/b)
Flip
Reverse the order of elements along dim. Materializing op
(not a view) — Layout’s strides are usize so the
negative-stride view path isn’t representable. Backward is
Flip { dim } itself (involutive).
Roll
Cyclic shift along dim by shift positions (positive
shifts move elements to higher indices, wrapping around).
Materializing op. Backward is Roll { dim, -shift }.
CumSum
Running cumulative sum along dim. Same shape as input.
Backward: reverse cumsum, expressed as Flip → CumSum → Flip
on the same dim using existing primitives.
Triu
Upper triangular: keep x[..., i, j] when j >= i + diagonal,
zero otherwise. Operates on the last two dims; leading dims
are batched. diagonal selects which diagonal is the cutoff
(0 = main diagonal, positive = above, negative = below).
Same shape as input. Backward is Triu { diagonal } itself
(linear mask; gradient passes through kept positions).
Tril
Lower triangular: keep x[..., i, j] when j <= i + diagonal,
zero otherwise. Operates on the last two dims; leading dims
are batched. diagonal selects which diagonal is the cutoff
(0 = main diagonal, positive = above, negative = below). The
canonical causal-attention mask is tril(diagonal = 0).
Same shape as input. Backward is Tril { diagonal } itself.
LogSoftmaxLastDim
Numerically-stable log-softmax along the last dimension:
y_i = x_i - max_j(x_j) - log(sum_j exp(x_j - max_j(x_j))).
Same shape as input. Used in NLL / cross-entropy loss where
the explicit log(softmax(x)) decomposition risks
over/underflow. Backward: see Op::LogSoftmaxLastDimBackward.
MaskedFill
MaskedFill: out = where(mask != 0, value, x). Inputs:
(x, mask) where mask is U8 (any nonzero counts as “fill”).
value is a scalar of the same dtype as x, stored on the
op. Output shape == x shape (mask must broadcast to x but the
simple-shape case requires identical shapes today). Backward
passes the upstream gradient through x at positions where
mask == 0, and contributes zero to mask (U8 anyway).
PadBackward
Backward helper for Op::Pad. Single input is the upstream
gradient (shape == padded forward output shape); output is the
gradient with respect to the original input (shape ==
in_shape). Mode determines accumulation behavior:
- Constant: gradient at padded slots is dropped; the unpadded region is the input gradient.
- Reflect / Replicate: gradient accumulates from output positions that map back to each input position via the forward index function.
Carries in_shape so the kernel can size the output tensor
(Op output shape isn’t enough — Op::Pad’s output shape
derivation needs reversing).
Pad
Multi-dim Pad: per-axis (before, after) extends each
dimension. Output shape: out[i] = in[i] + padding[i].0 + padding[i].1.
mode selects the fill behavior; value is the fill
constant for PadMode::Constant (ignored otherwise — see
the variant docs for those modes).
Multi-dim shape is the form PyTorch’s F.pad exposes — a
single call covers e.g. “pad an image by 1 px on every side”
(padding = [(0,0), (0,0), (1,1), (1,1)] for [N,C,H,W]).
Single-dim is just the special case padding[i] = (0,0)
everywhere except one axis.
Only Constant mode is fully implemented in v1. Reflect / Replicate are accepted at the IR level but the executor returns a clean “not yet implemented” error.
Backward (Constant): slice along each padded axis to drop the padded regions, restoring the input shape.
Where
Ternary select: out[i] = if cond[i] != 0 { a[i] } else { b[i] }.
Inputs (cond, a, b): cond is U8 (typically the output of
a comparison op), a and b share dtype T and shape with
cond. Output dtype = T, shape = cond.shape().
Differentiable through a and b (gradient flows only to
the picked positions); cond is non-differentiable. Backward
rule expressed via Op::Cast + scalar arithmetic on existing
primitives, registered in crate::grad::WhereRule.
MatMul
Rank-2 matrix multiply.
Transpose
Rank-2 transpose.
Permute(Vec<usize>)
N-dimensional permutation. The parameter is the new axis order:
out.shape[i] = in.shape[axes[i]]. For a rank-3 input [a, b, c]
with axes = [2, 0, 1], output shape is [c, a, b]. The axes
vector must be a permutation of 0..rank.
Cast(DType)
Convert a tensor to a different dtype. The target dtype is carried on the op itself; the source dtype is whatever the input node’s dtype is.
BroadcastTo(Shape)
Broadcast a tensor to a larger shape. The input shape must be
broadcast-compatible with the target (NumPy rules). Used directly
by user code and as the backward rule for SumAll, MeanAll,
and axis reductions.
Reshape(Shape)
Reshape a tensor to a new shape with the same element count. Data
is unchanged; only the shape metadata is replaced. Serves as the
general building block for arbitrary shape changes;
see Op::Unsqueeze for the metadata-only-view variant that
inserts a size-1 dim without ever materializing.
Contiguize
Materialize a contiguous copy of the input. Output shape and dtype match the input; only the layout changes (strides become row-major, start_offset becomes 0). Zero-cost when the input is already contiguous + zero-offset — the executor adopts the input’s Storage Arc unchanged in that case.
First-class IR concern so the optimizer (Phase 2.2) can insert
layout-fixups before kernels that don’t advertise
crate::KernelCaps::strided_input without overloading
Op::Reshape’s “change shape” semantics. The executor
compiles this to the same WorkItemKind::ContiguizeOf arm
Op::Reshape uses; the only difference is that
Op::Contiguize’s output shape equals its input shape, so
the executor’s element-count sanity check is trivial.
Unsqueeze
Insert a size-1 dimension at position dim (range 0..=rank,
where dim == rank appends to the end). Strictly more efficient
than Op::Reshape for the size-1-insertion case: this is a
metadata-only view op that shares bytes with its input via the
Layout side-table, so it preserves any upstream
strided/transposed/broadcast layout instead of triggering an
auto-Contiguize.
Squeeze
Drop a size-1 dimension at position dim (range 0..rank).
Inverse of Op::Unsqueeze; metadata-only view that shares
bytes with its input via the Layout side-table. Panics at build
time if the dim’s size isn’t 1 (mirrors Unsqueeze’s bounds
check — graph-builder validation, not runtime).
ReduceSumTo(Shape)
Sum-reduce a tensor to a smaller shape by summing along any dims
where the source was broadcast against the target. This is the
backward rule for BroadcastTo and is symmetric with it: both ops
are each other’s gradient. For a source shape [2, 3, 4] being
reduced to [3, 4], the sum is along dim 0; for [2, 3, 4] to
[2, 1, 4], the sum is along dim 1 while keeping the dim.
ReduceMaxTo(Shape)
Max-reduce a tensor to a smaller shape, the maximum-symmetric
counterpart of Op::ReduceSumTo. The target shape must be
broadcast-compatible into the source: along any axis where the
padded target is 1 (or absent), the input is reduced via max;
other axes carry through. Used by lowering rules (notably
SoftmaxLastDim’s max-subtract step) and as the keepdim form of
per-axis max reductions.
Backward through max-selection routes the upstream gradient
only to the argmax position, which requires holding onto the
original input and recomputing the argmax. PR 3.5 ships this
op without a backward rule (panics cleanly on .backward(),
matching Op::ArgMaxDim’s precedent); add the backward when a
real consumer needs it.
SumAll
Sum of every element, producing a rank-0 tensor.
MaxAll
Maximum of every element, producing a rank-0 tensor.
MinAll
Minimum of every element, producing a rank-0 tensor.
MeanAll
Arithmetic mean of every element, producing a rank-0 tensor.
SumDim(usize)
Sum along the given dim; the reduced dim is removed from the output.
MaxDim(usize)
Max along the given dim; the reduced dim is removed from the output.
MinDim(usize)
Min along the given dim; the reduced dim is removed from the output.
MeanDim(usize)
Mean along the given dim; the reduced dim is removed from the output.
LogSoftmaxLastDimBackward
LogSoftmax-last-dim backward. Inputs: (forward_log_softmax_output, upstream). Output: the gradient of the input to log-softmax.
Formula: g - exp(y) * sum(g, last_dim, keepdim=true) where
y is the forward log-softmax output and g is the upstream
gradient. Independent of Op::SoftmaxLastDimBackward because
the closed-form rule uses exp(y) = softmax(x) directly, not
the upstream-times-softmax composition.
ArgMaxDim(usize)
Index of the maximum along dim. Output: U32 with the reduced
dim removed.
ArgMinDim(usize)
Index of the minimum along dim. Output: U32 with the reduced
dim removed.
Concat
Concatenate two tensors along dim. Both must have the same rank,
same dtype, and equal sizes in every dim except dim.
Slice
Slice (narrow) a tensor along dim: take elements [start, start+len).
Output shape: same as input but with dim shrunk to len.
AddScalar(f64)
Add a scalar to every element.
MulScalar(f64)
Multiply every element by a scalar.
PowI(i32)
Raise every element to an integer power via repeated multiplication.
Works without an extra log/exp pair so it’s numerically clean
for small integer exponents like squaring or cubing.
Clamp
Clamp every element to the inclusive range [min, max].
Maximum
Element-wise maximum of two tensors with matching shape.
Minimum
Element-wise minimum of two tensors with matching shape.
IndexSelect
Index-select along a single dimension with a 1-D index vector.
Output shape: same as input, but with dim replaced by
indices.elem_count().
Gather
N-dimensional gather along dim. Output shape matches the index
tensor’s shape. out[..., k, ...] = data[..., indices[..., k, ...], ...]
with dim being the position where the index value substitutes
for the output coordinate.
IndexAdd
Index-add: the backward of IndexSelect. Takes a base tensor,
a 1-D U32 index vector, and a src tensor, and adds src to base
at the indexed positions along dim. out[..., indices[i], ...] += src[..., i, ...]. Non-indexed positions of base are copied
through unchanged. Works functionally (returns a new tensor).
ScatterAdd
Scatter-add: the backward of Gather. Takes a base tensor, an
N-D U32 index tensor matching the src shape, and a src tensor,
and accumulates src values into base at positions given by
substituting indices at position dim. out[p with dim ← indices[p]] += src[p].
NonZeroIndices
Data-determined nonzero-index extraction — the keystone primitive for data-dependent dynamic shapes (SSM autoregressive decode, MoE sparse dispatch, MLA compressed-KV). Given an input value tensor, produces the flat indices of its nonzero elements over a fixed-capacity buffer plus the true runtime count, using the multi-output bundle mechanism ([12-multi-output]):
- slot 0 (primary):
indices, shape[capacity], dtypeU32, wherecapacity = input.elem_count()(the static worst-case bound). The firstcountentries are the flat (row-major) indices of the input’s nonzero elements in ascending order; the tail[count..capacity]is unspecified padding. - slot 1:
count, shape[1], dtypeU32— the number of nonzero elements found.
“Nonzero” means != 0 in the input dtype’s arithmetic (a 0.0
float, an all-zero integer). The runtime count is published into
the per-pass SymEnv under count_sym by the
executor after this op realizes, so downstream ops consume it as
a DynScalar::Sym / dynamic Extent —
the same host-scalar-extent pattern the KV-cache cached_len
uses. This is the first op whose output length is determined by
its input data, not by build-time shapes (the fuel_ir::symbol::SymEnv bind is
the one net-new realize-time seam). Primitive (in the base map by
construction — no decompose). Non-differentiable: discrete
indices, backward drops gradient like Op::Gather’s index path.
Copy
Copy the input tensor to a specific device. Source stays
resident (non-destructive). Source device is implicit from
inputs[0]’s current residency. When source and target
devices match, the backend is free to turn this into a cheap
clone; when they differ, the backend must actually transfer
the data (today: through a host buffer — Phase 3 has no P2P
fast path yet).
Phase 3 of the unified scheduler work. The scheduler (Phase 4)
inserts these nodes automatically; users can also construct
them explicitly via NodeHandle::copy_to_device.
Fields
target: DeviceLocationRelease
Release the input tensor’s device-resident storage. Produces a zero-element marker output (NodeId placeholder for graph bookkeeping; no meaningful data). After this op runs, the input’s device storage is gone; any subsequent reader of the input’s NodeId must get its data from a different path (a sibling Copy output, or re-execution of the producer).
Destructive on inputs[0]. Scheduler must pin Op::Release
to run after every other reader of its input via
opt::derive_ordering.
Multi-output bundles (Option C, Session 3): when
inputs[0] is a multi-output producer (its Storage carries a
fuel_ir::storage::OutputView bundle), the bundle
is the single eviction unit — the Release evicts the whole
bundle, not a single slot. opt::collect_alias_set treats
every Op::View of the producer as part of the producer’s
alias set, so derive_ordering pins Release after every
reader of every View; the bundle drops only when the LAST
View consumer finishes. Op::ViewOwned consumers may run
freely after Release: their forward memcpy ran before the
Release fires (data-dep edge), and the resulting standalone
Storage is independent of the producer’s bundle. Per-slot
eviction (releasing one slot’s bytes while another stays
live) is intentionally a follow-up — v1 keeps the bundle as
a single unit.
Move
Transfer the input tensor to target device, destroying the
source in the process. Semantically equivalent to
Op::Copy { target } followed by Op::Release on the source,
with the explicit guarantee that backends MAY fast-path the
fused form as a single transfer (e.g., a zero-copy handoff if
source and target are the same memory type).
Output is a fresh tensor on target with the input’s shape
and dtype. Destructive on inputs[0]; the scheduler pins this
op to run after every non-destructive reader of the source via
opt::derive_ordering.
Fields
target: DeviceLocationWriteSlice
In-place scatter write: copies inputs[1]’s bytes into a
rectangular slab of inputs[0]’s storage, defined per-dim by
ranges[i] = (start, end). After the op runs, inputs[0]’s
bytes inside the slab are replaced; bytes outside the slab are
untouched.
Shape contract: inputs[0].rank() == inputs[1].rank() == ranges.len(). For each axis i,
inputs[1].dims()[i] == ranges[i].1 - ranges[i].0, and
ranges[i].1 <= inputs[0].dims()[i]. Dtypes must match.
Output is a marker that adopts inputs[0]’s Storage Arc and
Layout — same shape, same bytes, post-write. Consumers that
want a sub-extent of the destination compose an explicit
Op::Slice after the write (WriteSlice does not encode
post-write extents).
Destructive on inputs[0] (the destination is consumed; only
the write op’s output NodeId may read its bytes afterward).
Non-destructive on inputs[1]. The scheduler pins this op to
run after every other reader of the destination via
opt::derive_ordering.
Phase E.3.2: introduced to back persistent KV-cache writes
(InferenceContext + KvCache). Non-differentiable; backward
panics — KV-cache writes are forward-only.
dyn_offset (Phase D symbolic extents): None ⇒ fully static
(ranges alone defines the slab — today’s behavior). Some((axis, off)) ⇒ the destination start on axis is off.resolve(env)
against the per-pass fuel_ir::SymEnv at realize,
overriding ranges[axis].0; the slab width on that axis stays
ranges[axis].1 - ranges[axis].0. This is the input-determined
dynamic-offset path (a DynScalar over the one SymEnv),
distinct from Op::WriteSliceRotating’s data-determined
position (a tensor input). It backs the persistent decode
KV-cache write, whose append offset (cached_len) is a per-token
runtime value over a fixed-capacity buffer, so the graph
structure stays identical across tokens and the plan is reused.
Build a node with this set via NodeHandle::write_slice_dyn; the
resolved offset + width is bounds-checked against the
destination capacity at realize (a typed error, never a panic).
WriteSliceRotating
Like Op::WriteSlice but the axis axis wraps modulo
modulus. The dynamic write position comes through inputs[2]
(rank-0 U32); at realize time the kernel computes
start = position % modulus and writes inputs[1] into
inputs[0] at the wrapped offset, splitting across the
boundary if start + write_len > modulus.
Inputs:
inputs[0]: destination tensor (the rotating buffer). Shape onaxisis the full storage extent — typically equal tomodulus.inputs[1]: source slice. Shape on non-rotating axes is the slab inranges; shape onaxisis the write length.inputs[2]: dynamic write position, rank-0U32. Wrapped modulomodulusinside the kernel — callers pass the monotonic logical position (token index, etc.).
ranges describes the destination slab on every axis. For the
rotating axis axis, ranges[axis].0 is ignored (start is
dynamic) and ranges[axis].1 - ranges[axis].0 must equal
inputs[1].dims()[axis] (the write length). For other axes
the same shape contract as Op::WriteSlice applies.
Output is a marker that adopts inputs[0]’s Storage Arc and
Layout, post-write. Destructive on inputs[0]; scheduled like
Op::WriteSlice.
Phase C of the eager-NodeHandle retirement program. Backs sliding-window KV caches (Mistral / Phi-3 sliding-window / sliding-window Qwen). Non-differentiable; backward panics.
WriteSliceDoff
Like Op::WriteSlice but the start offset on axis is a
device-resident scalar read from inputs[2] (rank-0 I64)
at kernel-launch time — NOT host-resolved. This is the CUDA-
graph-capturable append: under capture, the offset buffer is
updated by a per-token fixed-address H2D memcpy before each
replay, so the captured graph is reused across decode steps
while the write lands at the live position. No modulo wrap
(unlike Op::WriteSliceRotating); the offset is the raw
start (cached_len).
Inputs:
inputs[0]: destination tensor (the fixed-capacity buffer).inputs[1]: source slab. Shape on non-axisaxes is the slab inranges; shape onaxisis the write width.inputs[2]: device offset, rank-0I64— the start onaxis. Overridesranges[axis].0(which is a placeholder ignored by the kernel).
ranges describes the destination slab on every axis. For the
dynamic axis, ranges[axis].0 is ignored (start is device-
resident) and ranges[axis].1 - ranges[axis].0 must equal
inputs[1].dims()[axis] (the write width). For other axes the
same shape contract as Op::WriteSlice applies.
Bounds are the caller’s contract: the kernel reads the start
device-side and does NOT clamp (clamping would perturb the index
math vs the static path). offset + width <= dest_dims[axis]
must hold — guaranteed by the caller sizing the buffer’s
capacity (DecodeSession sizes max_seq).
Output adopts inputs[0]’s Storage Arc + Layout, post-write.
Destructive on inputs[0]; scheduled like Op::WriteSlice.
Non-differentiable; backward panics.
CapturedRun (the decode-latency lever, 2026-07): the device-
resident-offset WriteSlice is form-B of baracuda’s
write_slice_*_doff launchers.
Alloc
Allocate a fresh, zero-initialized Storage of the node’s shape +
dtype on target. Zero inputs — Op::Alloc is a source op
like Op::Const, but its bytes are computed (zeros) rather
than seeded from the host.
Bridge-retirement Phase 3a (post-9c). Replaces the per-device-
location match in fuel-tensor::inference_context::alloc_zeroed_on
with a graph-level node the optimizer can see + the executor’s
WorkItemKind::Alloc arm dispatches per-backend.
Device-handle requirement: the executor’s Alloc arm derives
the per-backend handle (e.g. &CudaDevice, &Arc<VulkanBackend>)
by searching the input cache for any Storage already on
target’s backend. Callers must seed the cache with at least
one such storage before realizing an Op::Alloc on a non-CPU
target. fuel-tensor::pipelined_bridge::device_seed_storage is
the canonical seeder.
Fields
target: DeviceLocationZeroFill
Fill the input tensor’s bytes with zero, in place. Aliases
inputs[0]’s Storage Arc as the output (same Storage; the
bytes are mutated). Destructive on inputs[0] — readers of
the input’s NodeId after this op see post-fill bytes; new
readers should use this op’s NodeId.
Bridge-retirement Phase 3a follow-up. Paired with
Op::Alloc to give the architecturally clean “alloc uninit +
explicit zero” pipeline. Backends with native device-side
fills (CUDA cuMemsetD8Async, Vulkan vkCmdFillBuffer)
dispatch through their kernels; CPU does an
Arc::make_mut-style in-place memset.
Future extension: an Op::Fill { value: u8 } (or Scalar)
for non-zero patterns. ZeroFill is the only variant today
because zero is the only init pattern any current caller
needs.
Fused(FusedOpId, FusedOpParams)
Phase 7.6 single-arm delegate to the open
crate::registry::FusedOpRegistry. The id selects which
fused op (SoftmaxLastDim, RmsNormLastDim, FlashAttn, …) and
the params carry per-instance data. The registry’s metadata
side (decompose, backward identity, shape/dtype rules) lives in
fuel-graph::registry; the kernel side (per-backend KernelRef
- cost + precision) lives in
fuel-storage::fused.
Adding a new fused op is one registry entry plus one kernel
function per backend that supports it — no new Op variant.
Step 3 of Phase 7.6 migrates SoftmaxLastDim through this arm
as the proof of concept.
View
Project one logical output out of a multi-output producer node.
The producer’s realized Storage carries a
fuel_ir::storage::OutputView side-table; this op
reads slot slot of that side-table and exposes its
dtype/shape/layout as the View node’s output.
Single input (the multi-output producer). Output shares the
producer’s Arc<RwLock<Storage>> — zero bytes copied. The
bundle stays alive as long as any View consumer (or the
original producer handle) holds a clone.
At graph-build time, the View’s shape + dtype + layout come
from the producer’s slot specs recorded via
Graph::set_output_views; building a View { slot } over a
producer that hasn’t declared slot specs is a typed error
surfaced by the View-emitting builder helper. The bundle’s
per-slot dtype is independent of the producer node’s own
Node::dtype (which is the bundle’s primary/slot-0 dtype).
Backward (see fuel-graph::grad): the per-slot upstream
gradient is scattered into a zero-filled bundle of the
producer’s overall byte shape; the producer’s own backward
rule then sees a “bundle gradient” as its upstream. Until a
real multi-output op author exists (Session 2), the backward
rule emits an Op::Const zero of the producer’s primary
shape — the scatter-into-slot primitive lands alongside the
first differentiable multi-output op.
ViewOwned
Like Op::View, but copies the slot’s bytes into a fresh
standalone Storage. Costs one allocation + one slot-sized
memcpy; the producer’s bundle Arc can drop as soon as every
outstanding Op::View over the same producer also drops.
The planner promotes a View to ViewOwned when a slot’s
liveness substantially outlasts the rest of the bundle (the
classic case: Mamba’s last_state slot retained across an
autoregressive step while y is consumed immediately). v1
emits the same dispatch shape as Op::Copy — backend-side it’s
Copy with byte offset + length pulled from the slot spec.
Same shape/dtype/layout inference path as View; same backward
shape too (the forward already paid the copy).
ScatterIntoSlot
Multi-output autograd primitive (Option C, item 4). Takes two inputs:
inputs[0]: a bundled tensor (typically a zero-bundle)inputs[1]: the slot’s gradient
Produces a new bundled tensor identical to inputs[0] except
that slot slot’s byte range is overlaid by inputs[1]’s
bytes. The output is alias-free with respect to inputs[0]
(a fresh Storage with the new bytes) so multiple
ScatterIntoSlot operations on the same bundle_zero chain
sequentially without aliasing pitfalls.
Emitted by Op::View / Op::ViewOwned backward rules: each
View consumer scatters its upstream gradient into a fresh
zero-bundle, then the producer’s backward receives the
composed bundle gradient as its upstream.
Status: IR-level primitive only in this session. No CPU
kernel registration yet. The production multi-output SSM ops
(SelectiveScan, SsdChunkScan) are now differentiable via the
Op::Scan Phase-2 lower_scans_for_backward pre-pass (decompose →
Op::Scan → unroll_scan → node-general autograd), which lowers
them to primitives BEFORE the reverse walk — so the autograd
differentiates the unrolled View projections directly and never
reaches a ScatterIntoSlot realization on the SSM path.
Branch
In-place multi-path (phi/merge) node — the arena representation of the optimized form’s divergent-then-reconvergent routes.
A Branch node’s divergent “arms” are encoded in the node’s
existing inputs: Vec<NodeId>: each input is the exit of one
candidate route, and reconverge_at names the explicit node at
which those routes merge back into a single value. Because the
multi-path structure is an arena fact rather than an overlay, it
persists, compacts, graph-walks, and path-filters exactly like
any other node; a graph with zero Op::Branch nodes is
exactly today’s single-route graph (back-compatible by
construction).
Inert until PR-A1. This variant is the PR-A0 scaffold: the
closed Op enum carries it and every exhaustive match handles
it, but nothing constructs it yet. The open_branch /
add_arm / finalize_branches builders (with build-time
validation: reconverge must descend from the diverge point, arms
internally disjoint, cast-to-uniform at reconverge) land in
PR-A1. Until then, accessors return a constant (“branch”) and
fallible paths (shape inference, autograd, lowering) return a
descriptive Err rather than panicking.
Scan
Fuel’s first sub-graph-carrying primitive: a bounded lax.scan-shaped
recurrence scan(body, init_carry, xs, consts) -> (final_carry, ys).
The body is encoded in this node’s own inputs, exactly like
Op::Branch’s arms. Node layout:
inputs = [ init_carry_0..init_carry_{n_carries-1}, xs_0..xs_{n_xs-1}, consts..., body_new_carry_0..body_new_carry_{n_carries-1}, body_y ]
— the two body-exit NodeIds are the LAST two inputs, so base_map_hash,
topo_order_multi, and reachability see the body for free. The body
references Op::ScanPlaceholder{Carry,0} for the per-step carry and
Op::ScanPlaceholder{Elem,i} for the per-step slice of xs[i]; consts
are referenced by real NodeId. n_carries carries are threaded; the
body references ScanPlaceholder{Carry,k} for carry k (GAP-303 —
n_carries = 1 is the v1 single-carry model).
An optimization/verification-time node: it never reaches the executor
un-lowered (the SSM decompose emits it, but the executor dispatches the
fused op to its kernel; unroll_scan materializes primitives on demand
for the verify oracle / kernel-absent fallback). No LoweringRule
matches it — it is a terminal in the base map, which is correct: it IS
the primitive. Always a 2-slot bundle (slot 0 = stacked ys, slot 1 =
final carry). early_exit is Phase-2 (see ScanPredicate).
Fields
n_carries: usizeHow many recurrent carries this scan threads. Symmetric with
n_xs: the carry NODES live in inputs (a positional block, see
the layout above); this field is only their COUNT, exactly as
n_xs is only the count of the xs block.
GAP-303: 1 reproduces the v1 single-carry model exactly.
early_exit: Option<ScanPredicate>ScanPlaceholder
Inert body-hole leaf for Op::Scan (no inputs). Substituted by
unroll_scan per step. Never directly realized.
Implementations§
Source§impl Op
impl Op
Sourcepub fn destructive_input(&self) -> Option<usize>
pub fn destructive_input(&self) -> Option<usize>
Index into inputs that this op destroys on execution. None
means the op is non-destructive — every input remains readable
after the op completes. Destructive ops need the scheduler to
pin them to run after all other readers of the destroyed input,
via ordering edges derived by opt::derive_ordering.
Sourcepub fn short_name(&self) -> &'static str
pub fn short_name(&self) -> &'static str
Short human-readable name for this op — used by executor error
messages to identify the offending graph node without spilling
all of Debug’s field contents. Keeps panic messages one-liner-
friendly while still telling you which kind of op blew up.
Sourcepub fn is_view_op(&self) -> bool
pub fn is_view_op(&self) -> bool
Whether this op is a metadata-only view: its output shares bytes with its single input, reinterpreting them through new strides and/or offset. Executors realize these without allocating new storage — they wrap the input’s Storage in a fresh Layout.
Note that Op::Reshape is not a view op. Reshape produces
contiguous output: zero-copy when its input is contiguous, but
it materializes via auto-Contiguize when the input is strided.
The other shape-altering ops in this set are always view-shaped
regardless of the input layout.