pub struct Variable { /* private fields */ }Expand description
A differentiable variable wrapping a Tensor.
Variables use libtorch’s native autograd. When a tensor has
requires_grad=true, all standard operations build a C++ computation
graph automatically. Calling backward() runs libtorch’s backward
engine — no Rust-side graph walking.
Internally uses Rc<RefCell<>> for shared ownership — the same
parameter can be referenced by both a Module and an Optimizer.
let w = Variable::new(Tensor::randn(&[3, 2], opts)?, true);
let x = Variable::new(Tensor::from_f32(&[1.0, 2.0, 3.0], &[1, 3], Device::CPU)?, false);
let loss = x.matmul(&w)?.sum()?;
loss.backward()?;
println!("{:?}", w.grad()); // gradient of loss w.r.t. wImplementations§
Source§impl Variable
impl Variable
Sourcepub fn mul(&self, other: &Variable) -> Result<Variable, TensorError>
pub fn mul(&self, other: &Variable) -> Result<Variable, TensorError>
Element-wise multiplication (Hadamard product).
Sourcepub fn matmul(&self, other: &Variable) -> Result<Variable, TensorError>
pub fn matmul(&self, other: &Variable) -> Result<Variable, TensorError>
Matrix multiplication (torch.matmul). Supports batched matmul.
Sourcepub fn mul_scalar(&self, scalar: f64) -> Result<Variable, TensorError>
pub fn mul_scalar(&self, scalar: f64) -> Result<Variable, TensorError>
Multiply all elements by a scalar.
Sourcepub fn div_scalar(&self, scalar: f64) -> Result<Variable, TensorError>
pub fn div_scalar(&self, scalar: f64) -> Result<Variable, TensorError>
Divide all elements by a scalar.
Sourcepub fn add_scalar(&self, scalar: f64) -> Result<Variable, TensorError>
pub fn add_scalar(&self, scalar: f64) -> Result<Variable, TensorError>
Add a scalar to all elements.
Sourcepub fn neg(&self) -> Result<Variable, TensorError>
pub fn neg(&self) -> Result<Variable, TensorError>
Element-wise negation.
Sourcepub fn relu(&self) -> Result<Variable, TensorError>
pub fn relu(&self) -> Result<Variable, TensorError>
ReLU activation: max(0, x).
Sourcepub fn sigmoid(&self) -> Result<Variable, TensorError>
pub fn sigmoid(&self) -> Result<Variable, TensorError>
Sigmoid activation: 1 / (1 + exp(-x)).
Sourcepub fn tanh(&self) -> Result<Variable, TensorError>
pub fn tanh(&self) -> Result<Variable, TensorError>
Tanh activation: element-wise hyperbolic tangent.
Sourcepub fn gelu(&self) -> Result<Variable, TensorError>
pub fn gelu(&self) -> Result<Variable, TensorError>
GELU activation (Gaussian Error Linear Unit, erf form):
0.5 * x * (1 + erf(x / sqrt(2))).
Sourcepub fn gelu_tanh(&self) -> Result<Variable, TensorError>
pub fn gelu_tanh(&self) -> Result<Variable, TensorError>
Tanh-approximation GELU. Matches HuggingFace hidden_act="gelu_new"
and PyTorch F.gelu(x, approximate="tanh"). Used by ALBERT, GPT-2,
and any HF checkpoint that ships hidden_act in {gelu_new,
gelu_pytorch_tanh}.
Sourcepub fn silu(&self) -> Result<Variable, TensorError>
pub fn silu(&self) -> Result<Variable, TensorError>
SiLU activation (Sigmoid Linear Unit, aka Swish): x * sigmoid(x).
Sourcepub fn leaky_relu(&self, negative_slope: f64) -> Result<Variable, TensorError>
pub fn leaky_relu(&self, negative_slope: f64) -> Result<Variable, TensorError>
Leaky ReLU: max(0, x) + negative_slope * min(0, x).
Sourcepub fn elu(&self, alpha: f64) -> Result<Variable, TensorError>
pub fn elu(&self, alpha: f64) -> Result<Variable, TensorError>
ELU: max(0, x) + min(0, alpha * (exp(x) - 1)).
Sourcepub fn softplus(
&self,
beta: f64,
threshold: f64,
) -> Result<Variable, TensorError>
pub fn softplus( &self, beta: f64, threshold: f64, ) -> Result<Variable, TensorError>
Softplus: (1/beta) * log(1 + exp(beta * x)).
Sourcepub fn mish(&self) -> Result<Variable, TensorError>
pub fn mish(&self) -> Result<Variable, TensorError>
Mish: x * tanh(softplus(x)).
Sourcepub fn selu(&self) -> Result<Variable, TensorError>
pub fn selu(&self) -> Result<Variable, TensorError>
SELU: self-normalizing ELU with fixed alpha/lambda.
Sourcepub fn hardswish(&self) -> Result<Variable, TensorError>
pub fn hardswish(&self) -> Result<Variable, TensorError>
Hardswish: x * clamp(x + 3, 0, 6) / 6.
Sourcepub fn hardsigmoid(&self) -> Result<Variable, TensorError>
pub fn hardsigmoid(&self) -> Result<Variable, TensorError>
Hardsigmoid: clamp(x + 3, 0, 6) / 6.
Sourcepub fn prelu(&self, weight: &Variable) -> Result<Variable, TensorError>
pub fn prelu(&self, weight: &Variable) -> Result<Variable, TensorError>
PReLU: max(0, x) + weight * min(0, x).
Sourcepub fn sum(&self) -> Result<Variable, TensorError>
pub fn sum(&self) -> Result<Variable, TensorError>
Sum of all elements, returning a scalar.
Sourcepub fn mean(&self) -> Result<Variable, TensorError>
pub fn mean(&self) -> Result<Variable, TensorError>
Mean of all elements, returning a scalar.
Sourcepub fn sum_dim(&self, dim: i32, keepdim: bool) -> Result<Variable, TensorError>
pub fn sum_dim(&self, dim: i32, keepdim: bool) -> Result<Variable, TensorError>
Sum along a dimension. If keepdim, the reduced dimension is retained with size 1.
Sourcepub fn mean_dim(&self, dim: i32, keepdim: bool) -> Result<Variable, TensorError>
pub fn mean_dim(&self, dim: i32, keepdim: bool) -> Result<Variable, TensorError>
Mean along a dimension. If keepdim, the reduced dimension is retained with size 1.
Sourcepub fn prod(&self) -> Result<Variable, TensorError>
pub fn prod(&self) -> Result<Variable, TensorError>
Product of all elements, returning a scalar.
Sourcepub fn prod_dim(&self, dim: i32, keepdim: bool) -> Result<Variable, TensorError>
pub fn prod_dim(&self, dim: i32, keepdim: bool) -> Result<Variable, TensorError>
Product along a dimension.
Sourcepub fn cumsum(&self, dim: i32) -> Result<Variable, TensorError>
pub fn cumsum(&self, dim: i32) -> Result<Variable, TensorError>
Cumulative sum along a dimension.
Sourcepub fn logsumexp(
&self,
dim: i32,
keepdim: bool,
) -> Result<Variable, TensorError>
pub fn logsumexp( &self, dim: i32, keepdim: bool, ) -> Result<Variable, TensorError>
Log of summed exponentials along a dimension (numerically stable).
Sourcepub fn min(&self) -> Result<Variable, TensorError>
pub fn min(&self) -> Result<Variable, TensorError>
Minimum of all elements, returning a scalar.
Sourcepub fn max(&self) -> Result<Variable, TensorError>
pub fn max(&self) -> Result<Variable, TensorError>
Maximum of all elements, returning a scalar.
Sourcepub fn min_dim(&self, dim: i32, keepdim: bool) -> Result<Variable, TensorError>
pub fn min_dim(&self, dim: i32, keepdim: bool) -> Result<Variable, TensorError>
Minimum values along a dimension.
Sourcepub fn max_dim(&self, dim: i32, keepdim: bool) -> Result<Variable, TensorError>
pub fn max_dim(&self, dim: i32, keepdim: bool) -> Result<Variable, TensorError>
Maximum values along a dimension.
Sourcepub fn var(&self) -> Result<Variable, TensorError>
pub fn var(&self) -> Result<Variable, TensorError>
Variance of all elements (Bessel-corrected).
Sourcepub fn std(&self) -> Result<Variable, TensorError>
pub fn std(&self) -> Result<Variable, TensorError>
Standard deviation of all elements (Bessel-corrected).
Sourcepub fn var_dim(&self, dim: i32, keepdim: bool) -> Result<Variable, TensorError>
pub fn var_dim(&self, dim: i32, keepdim: bool) -> Result<Variable, TensorError>
Variance along a dimension (Bessel-corrected).
Sourcepub fn std_dim(&self, dim: i32, keepdim: bool) -> Result<Variable, TensorError>
pub fn std_dim(&self, dim: i32, keepdim: bool) -> Result<Variable, TensorError>
Standard deviation along a dimension (Bessel-corrected).
Sourcepub fn softmax(&self, dim: i32) -> Result<Variable, TensorError>
pub fn softmax(&self, dim: i32) -> Result<Variable, TensorError>
Softmax along dim (output sums to 1).
Sourcepub fn log_softmax(&self, dim: i32) -> Result<Variable, TensorError>
pub fn log_softmax(&self, dim: i32) -> Result<Variable, TensorError>
Log-softmax along dim (numerically stable log(softmax(x))).
Sourcepub fn exp(&self) -> Result<Variable, TensorError>
pub fn exp(&self) -> Result<Variable, TensorError>
Element-wise exponential.
Sourcepub fn log(&self) -> Result<Variable, TensorError>
pub fn log(&self) -> Result<Variable, TensorError>
Element-wise natural logarithm.
Sourcepub fn sqrt(&self) -> Result<Variable, TensorError>
pub fn sqrt(&self) -> Result<Variable, TensorError>
Element-wise square root.
Sourcepub fn abs(&self) -> Result<Variable, TensorError>
pub fn abs(&self) -> Result<Variable, TensorError>
Element-wise absolute value.
Sourcepub fn pow_scalar(&self, exponent: f64) -> Result<Variable, TensorError>
pub fn pow_scalar(&self, exponent: f64) -> Result<Variable, TensorError>
Raise each element to a scalar power.
Sourcepub fn triu(&self, diagonal: i64) -> Result<Variable, TensorError>
pub fn triu(&self, diagonal: i64) -> Result<Variable, TensorError>
Upper triangular part. diagonal=0 keeps the main diagonal; positive shifts up.
Sourcepub fn tril(&self, diagonal: i64) -> Result<Variable, TensorError>
pub fn tril(&self, diagonal: i64) -> Result<Variable, TensorError>
Lower triangular part. diagonal=0 keeps the main diagonal; negative shifts down.
Sourcepub fn sin(&self) -> Result<Variable, TensorError>
pub fn sin(&self) -> Result<Variable, TensorError>
Element-wise sine.
Sourcepub fn cos(&self) -> Result<Variable, TensorError>
pub fn cos(&self) -> Result<Variable, TensorError>
Element-wise cosine.
Sourcepub fn sign(&self) -> Result<Variable, TensorError>
pub fn sign(&self) -> Result<Variable, TensorError>
Element-wise sign (-1, 0, or 1).
Sourcepub fn floor(&self) -> Result<Variable, TensorError>
pub fn floor(&self) -> Result<Variable, TensorError>
Element-wise floor (round toward negative infinity).
Sourcepub fn ceil(&self) -> Result<Variable, TensorError>
pub fn ceil(&self) -> Result<Variable, TensorError>
Element-wise ceil (round toward positive infinity).
Sourcepub fn round(&self) -> Result<Variable, TensorError>
pub fn round(&self) -> Result<Variable, TensorError>
Element-wise round to nearest integer.
Sourcepub fn reciprocal(&self) -> Result<Variable, TensorError>
pub fn reciprocal(&self) -> Result<Variable, TensorError>
Element-wise reciprocal (1/x).
Sourcepub fn clamp(&self, min: f64, max: f64) -> Result<Variable, TensorError>
pub fn clamp(&self, min: f64, max: f64) -> Result<Variable, TensorError>
Clamp all elements to [min, max].
Sourcepub fn clamp_min(&self, min: f64) -> Result<Variable, TensorError>
pub fn clamp_min(&self, min: f64) -> Result<Variable, TensorError>
Clamp all elements to be at least min.
Sourcepub fn clamp_max(&self, max: f64) -> Result<Variable, TensorError>
pub fn clamp_max(&self, max: f64) -> Result<Variable, TensorError>
Clamp all elements to be at most max.
Sourcepub fn log1p(&self) -> Result<Variable, TensorError>
pub fn log1p(&self) -> Result<Variable, TensorError>
Element-wise log(1 + x), numerically stable for small x.
Sourcepub fn expm1(&self) -> Result<Variable, TensorError>
pub fn expm1(&self) -> Result<Variable, TensorError>
Element-wise exp(x) - 1, numerically stable for small x.
Sourcepub fn log2(&self) -> Result<Variable, TensorError>
pub fn log2(&self) -> Result<Variable, TensorError>
Element-wise base-2 logarithm.
Sourcepub fn log10(&self) -> Result<Variable, TensorError>
pub fn log10(&self) -> Result<Variable, TensorError>
Element-wise base-10 logarithm.
Sourcepub fn atan2(&self, other: &Variable) -> Result<Variable, TensorError>
pub fn atan2(&self, other: &Variable) -> Result<Variable, TensorError>
Element-wise atan2 (arc tangent of y/x).
Sourcepub fn maximum(&self, other: &Variable) -> Result<Variable, TensorError>
pub fn maximum(&self, other: &Variable) -> Result<Variable, TensorError>
Element-wise maximum of two variables.
Sourcepub fn minimum(&self, other: &Variable) -> Result<Variable, TensorError>
pub fn minimum(&self, other: &Variable) -> Result<Variable, TensorError>
Element-wise minimum of two variables.
Sourcepub fn masked_fill(
&self,
mask: &Tensor,
value: f64,
) -> Result<Variable, TensorError>
pub fn masked_fill( &self, mask: &Tensor, value: f64, ) -> Result<Variable, TensorError>
Fill elements where mask is true (non-zero) with value.
Sourcepub fn normalize(&self, p: f64, dim: i32) -> Result<Variable, TensorError>
pub fn normalize(&self, p: f64, dim: i32) -> Result<Variable, TensorError>
L_p normalize along a dimension.
Sourcepub fn cosine_similarity(
&self,
other: &Variable,
dim: i64,
eps: f64,
) -> Result<Variable, TensorError>
pub fn cosine_similarity( &self, other: &Variable, dim: i64, eps: f64, ) -> Result<Variable, TensorError>
Cosine similarity between two variables along a dimension.
Sourcepub fn reshape(&self, shape: &[i64]) -> Result<Variable, TensorError>
pub fn reshape(&self, shape: &[i64]) -> Result<Variable, TensorError>
Return a view with a different shape. Use -1 for one inferred dimension.
Sourcepub fn transpose(&self, dim0: i32, dim1: i32) -> Result<Variable, TensorError>
pub fn transpose(&self, dim0: i32, dim1: i32) -> Result<Variable, TensorError>
Swap two dimensions.
Sourcepub fn permute(&self, dims: &[i64]) -> Result<Variable, TensorError>
pub fn permute(&self, dims: &[i64]) -> Result<Variable, TensorError>
Reorder dimensions according to the given permutation.
Sourcepub fn unsqueeze(&self, dim: i32) -> Result<Variable, TensorError>
pub fn unsqueeze(&self, dim: i32) -> Result<Variable, TensorError>
Insert a dimension of size 1 at dim.
Sourcepub fn unsqueeze_many(&self, dims: &[i32]) -> Result<Variable, TensorError>
pub fn unsqueeze_many(&self, dims: &[i32]) -> Result<Variable, TensorError>
Insert size-1 dimensions at multiple positions.
Sourcepub fn flatten(
&self,
start_dim: i32,
end_dim: i32,
) -> Result<Variable, TensorError>
pub fn flatten( &self, start_dim: i32, end_dim: i32, ) -> Result<Variable, TensorError>
Flatten dimensions from start_dim to end_dim (inclusive) into one.
Sourcepub fn expand(&self, shape: &[i64]) -> Result<Variable, TensorError>
pub fn expand(&self, shape: &[i64]) -> Result<Variable, TensorError>
Broadcast to shape without copying data. Use -1 to keep a dimension.
Sourcepub fn narrow(
&self,
dim: i32,
start: i64,
length: i64,
) -> Result<Variable, TensorError>
pub fn narrow( &self, dim: i32, start: i64, length: i64, ) -> Result<Variable, TensorError>
Slice along dim from start for length elements.
Sourcepub fn select(&self, dim: i32, index: i64) -> Result<Variable, TensorError>
pub fn select(&self, dim: i32, index: i64) -> Result<Variable, TensorError>
Select a single index along dim, removing that dimension.
Sourcepub fn index_select(
&self,
dim: i32,
index: &Tensor,
) -> Result<Variable, TensorError>
pub fn index_select( &self, dim: i32, index: &Tensor, ) -> Result<Variable, TensorError>
Select multiple indices along dim using a 1-D index tensor.
Sourcepub fn gather(&self, dim: i32, index: &Tensor) -> Result<Variable, TensorError>
pub fn gather(&self, dim: i32, index: &Tensor) -> Result<Variable, TensorError>
Gather values along dim using an index tensor of the same rank.
Sourcepub fn cat(&self, other: &Variable, dim: i32) -> Result<Variable, TensorError>
pub fn cat(&self, other: &Variable, dim: i32) -> Result<Variable, TensorError>
Concatenate two variables along dim.
Sourcepub fn cat_many(vars: &[&Variable], dim: i32) -> Result<Variable, TensorError>
pub fn cat_many(vars: &[&Variable], dim: i32) -> Result<Variable, TensorError>
Concatenate multiple variables along dim (torch.cat).
Sourcepub fn stack(vars: &[Variable], dim: i32) -> Result<Variable, TensorError>
pub fn stack(vars: &[Variable], dim: i32) -> Result<Variable, TensorError>
Stack variables along a new dimension (torch.stack).
Sourcepub fn chunk(&self, chunks: i32, dim: i32) -> Result<Vec<Variable>, TensorError>
pub fn chunk(&self, chunks: i32, dim: i32) -> Result<Vec<Variable>, TensorError>
Split into chunks pieces along dim (torch.chunk).
Sourcepub fn repeat(&self, repeats: &[i64]) -> Result<Variable, TensorError>
pub fn repeat(&self, repeats: &[i64]) -> Result<Variable, TensorError>
Repeat the tensor along each dimension by the given counts.
Sourcepub fn pad(&self, padding: &[i64], value: f64) -> Result<Variable, TensorError>
pub fn pad(&self, padding: &[i64], value: f64) -> Result<Variable, TensorError>
Constant-value padding. padding is [left, right, ...] from the last dim inward.
Source§impl Variable
impl Variable
Sourcepub fn new(data: Tensor, requires_grad: bool) -> Variable
pub fn new(data: Tensor, requires_grad: bool) -> Variable
Create a leaf variable (parameter or input data).
If requires_grad is true, libtorch will track operations for autodiff.
§Panics
Panics if gradient tracking cannot be enabled — e.g. on integer dtypes (libtorch: only floating-point tensors can require gradients). PyTorch raises the same error; silently returning a non-tracking variable would train nothing for this parameter.
Sourcepub fn data(&self) -> Tensor
pub fn data(&self) -> Tensor
Get the underlying tensor data (shallow clone sharing storage).
The returned Tensor shares the same memory as the variable’s data —
the same aliasing semantics as PyTorch’s .data. In-place mutations
on either side will be visible to both. If you need an independent
copy, call .data().copy() instead. Because the share counts as the
same tensor for concurrency purposes, the thread-safety rules on
Tensor apply across it: never
mutate in place while any other thread accesses the variable.
Sourcepub fn id(&self) -> usize
pub fn id(&self) -> usize
Stable identity of the shared autograd cell: clones of the same
Variable return the same id, independent copies differ. This
is what parameter collection dedups on (a module graph can reach
the same shared parameter through several paths). Valid only
while some clone is alive — ids can be recycled after the last
clone drops.
Sourcepub fn grad(&self) -> Option<Tensor>
pub fn grad(&self) -> Option<Tensor>
Get the accumulated gradient, if any. Reads from the C++ tensor’s .grad() field.
Sourcepub fn set_grad(&self, grad: Tensor)
pub fn set_grad(&self, grad: Tensor)
Replace the gradient tensor directly (e.g. for gradient clipping or unscaling).
Equivalent to param.grad = grad in PyTorch.
Panics if the FFI write fails (broken tensor handle): dropping the write silently would let the optimizer step with the old gradient.
Sourcepub fn requires_grad(&self) -> bool
pub fn requires_grad(&self) -> bool
Whether this variable tracks gradients.
Sourcepub fn set_requires_grad(&self, requires_grad: bool) -> Result<(), TensorError>
pub fn set_requires_grad(&self, requires_grad: bool) -> Result<(), TensorError>
Change whether this variable tracks gradients.
Replaces the inner data handle (the FFI returns a new handle sharing storage).
All clones of this Variable share the same Rc<RefCell>, so the change
is visible everywhere (module, optimizer, etc.).
Sourcepub fn is_leaf(&self) -> bool
pub fn is_leaf(&self) -> bool
Whether this is a leaf variable (no grad_fn in libtorch). A leaf tensor is one created by the user, not by an operation.
Sourcepub fn ensure_grad_accumulator(
&self,
) -> Result<Option<GradAccumulatorHandle>, TensorError>
pub fn ensure_grad_accumulator( &self, ) -> Result<Option<GradAccumulatorHandle>, TensorError>
Force creation of the AccumulateGrad node for a leaf variable
(requires_grad=true). The node’s stream is captured from the
current CUDA stream at the moment of this call. Returns a
handle that keeps the node alive; store it for the lifetime of
the worker.
Call this under a StreamGuard
on the training compute stream during DDP worker setup so that
gradient accumulation runs on the same stream as subsequent
forward/backward passes. Without this, the node is created
lazily on first backward() — inside the autograd engine’s
worker thread, whose current stream is the device default, and
libtorch fires the “AccumulateGrad node’s stream does not match”
warning on every run.
Returns Ok(None) for non-leaf or non-requires-grad variables.
Sourcepub fn autograd_node_count(&self) -> i64
pub fn autograd_node_count(&self) -> i64
Count unique autograd nodes reachable from this variable’s grad_fn. Returns 0 for leaf variables. Measures graph complexity — compare against Python’s equivalent to detect decomposed-op bloat.
Sourcepub fn item(&self) -> Result<f64, TensorError>
pub fn item(&self) -> Result<f64, TensorError>
Extract a scalar value as f64. The tensor must contain exactly one element.
Sourcepub fn zero_grad(&self)
pub fn zero_grad(&self)
Zero out the accumulated gradient (fills .grad() with zeros).
See also zero_grad_set_to_none for the faster alternative.
Panics if the FFI call fails (broken tensor handle): a silently skipped zero would leak the previous step’s gradient into the next.
Sourcepub fn zero_grad_set_to_none(&self)
pub fn zero_grad_set_to_none(&self)
Null out the gradient instead of zeroing it. No CUDA kernel.
Sourcepub fn detach(&self) -> Variable
pub fn detach(&self) -> Variable
Detach from the computation graph. Returns a new leaf variable sharing the same data tensor (detached) with no gradient tracking.
Panics if the FFI call fails (broken tensor handle): falling back to the attached tensor would silently keep gradients flowing where the caller asked to cut them.
Sourcepub fn to_device(&self, device: Device) -> Result<Variable, TensorError>
pub fn to_device(&self, device: Device) -> Result<Variable, TensorError>
Move to a different device. Returns a new leaf variable.
Sourcepub fn set_data(&self, data: Tensor)
pub fn set_data(&self, data: Tensor)
Replace the underlying tensor data (used by optimizers).
Preserves the requires_grad flag from the current tensor.
Bumps data_generation so caches built
from the old tensor can detect the replacement.
Sourcepub fn data_generation(&self) -> u64
pub fn data_generation(&self) -> u64
Monotonic generation of the underlying tensor: incremented every
time set_data replaces it (checkpoint load,
dtype cast, device move). In-place mutation (copy_, optimizer
steps) does not change it. Module caches built from parameter
tensors key their validity on this.
Sourcepub fn backward(&self) -> Result<(), TensorError>
pub fn backward(&self) -> Result<(), TensorError>
Run backward pass from this scalar variable. Populates .grad() on all leaf variables in the computation graph.
After backward completes, the tensor is detached in-place to immediately release the C++ grad_fn chain. Without this, the autograd Node objects stay alive until the Variable is dropped.