pub struct Param {
pub value: Matrix,
pub grad: Matrix,
/* private fields */
}Expand description
A weight matrix plus its gradient and Adam moments.
Only the value is serialized: a snapshot restores the weights and restarts
the moment estimates from zero, which is what
Network::load_json already does for
dense networks.
Fields§
§value: Matrix§grad: MatrixImplementations§
Source§impl Param
impl Param
pub fn new(value: Matrix) -> Self
pub fn zeros(rows: usize, cols: usize) -> Self
Sourcepub fn filled(rows: usize, cols: usize, value: f32) -> Self
pub fn filled(rows: usize, cols: usize, value: f32) -> Self
Every entry set to value. RMSNorm scales start at one.
Sourcepub fn he_uniform(
rows: usize,
cols: usize,
fan_in: usize,
rng: &mut StdRng,
) -> Self
pub fn he_uniform( rows: usize, cols: usize, fan_in: usize, rng: &mut StdRng, ) -> Self
He-style uniform initialization, matching the dense network’s scheme so
a transformer built with a given seed is as reproducible as a Network.
pub fn len(&self) -> usize
pub fn is_empty(&self) -> bool
Sourcepub fn freeze(&mut self)
pub fn freeze(&mut self)
Stops the parameter training and releases its gradient and Adam moments, which is three quarters of what it costs to train.
The weights stay, and a forward pass is unaffected. Reversed by
Param::unfreeze, which allocates the buffers again, empty.
pub fn is_frozen(&self) -> bool
Sourcepub fn set_fake_quantize(&mut self, enabled: bool)
pub fn set_fake_quantize(&mut self, enabled: bool)
Rounds this weight through the int8 grid in the forward pass, without changing what is stored or what the optimizer updates.
Sourcepub fn is_fake_quantized(&self) -> bool
pub fn is_fake_quantized(&self) -> bool
Whether the forward pass rounds this weight.
pub fn zero_grad(&mut self)
Sourcepub fn to_cuda(
&mut self,
device: usize,
memory_budget_mib: usize,
) -> Result<(), NetworkError>
pub fn to_cuda( &mut self, device: usize, memory_budget_mib: usize, ) -> Result<(), NetworkError>
Moves the parameter onto device and keeps it there until
Param::to_cpu.
While it is resident the device owns the weights: the host value,
grad and moments are stale, and Linear::forward and
Linear::backward run their GEMMs on the device.
A model with more than one parameter should open a CudaDevice once
and use Param::to_cuda_on instead; this builds a context, and a
context per weight is the thing that makes a two-hundred-parameter model
slow to start.
Fails closed: built without the cuda feature this returns
NetworkError::CudaFeatureDisabled rather than staying on the host
and saying nothing.
Sourcepub fn to_cuda_on(&mut self, device: &CudaDevice) -> Result<(), NetworkError>
pub fn to_cuda_on(&mut self, device: &CudaDevice) -> Result<(), NetworkError>
Param::to_cuda onto a device a caller already opened.
Sourcepub fn to_cpu(&mut self) -> Result<(), NetworkError>
pub fn to_cpu(&mut self) -> Result<(), NetworkError>
Copies the weights and gradient back from the device and drops the mirror. A parameter that was never on a device is left alone.
Sourcepub fn moments(&mut self) -> Result<(&Matrix, &Matrix), NetworkError>
pub fn moments(&mut self) -> Result<(&Matrix, &Matrix), NetworkError>
The Adam moments, refreshed from the device first when the parameter is resident there.
A snapshot carries weights only, so a run that stops and resumes has to
carry the moments separately; see
TransformerLm::save_optimizer_state.
Sourcepub fn set_moments(
&mut self,
first: Matrix,
second: Matrix,
) -> Result<(), NetworkError>
pub fn set_moments( &mut self, first: Matrix, second: Matrix, ) -> Result<(), NetworkError>
Restores moments saved by an earlier run, uploading them when the parameter is already resident on a device.
Sourcepub fn set_value(&mut self, value: Matrix) -> Result<(), NetworkError>
pub fn set_value(&mut self, value: Matrix) -> Result<(), NetworkError>
Replaces the weights, uploading them when the parameter is resident on a device.
The shape has to match: a weight is wired into a forward pass by its
dimensions, and a caller that meant to reshape wants a new parameter,
not this. Used by Ema::swap_in to stand an
averaged weight in the live one’s place.
Sourcepub fn grad_sum_squares(&self) -> Result<f64, NetworkError>
pub fn grad_sum_squares(&self) -> Result<f64, NetworkError>
The sum of the squared gradient entries, for a caller building a global gradient norm across every parameter.
Accumulated in f64: a 100M-parameter model sums that many small
squares into one number, and an f32 accumulator loses the tail of
them.
Sourcepub fn step(&mut self, optimizer: &Optimizer, step: usize, scale: f32)
pub fn step(&mut self, optimizer: &Optimizer, step: usize, scale: f32)
Applies one optimizer update and clears the gradient.
step is the global Adam step count, already incremented: the first
update is step 1, not 0 (while Schedule::rate counts from 0). Step 0
panics, because Adam’s bias correction 1 - beta^0 would divide by zero
and turn every parameter NaN. SGD does not read it. scale divides the
accumulated gradient, which lets a caller accumulate over a batch and
average once here.
Weight decay applies to parameters with more than one row only. A one-row parameter is a bias or a norm gain, and shrinking those towards zero only fights the model (the nanoGPT rule).
Trait Implementations§
Source§impl<'de> Deserialize<'de> for Param
impl<'de> Deserialize<'de> for Param
Source§fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
__D: Deserializer<'de>,
fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
__D: Deserializer<'de>,
Auto Trait Implementations§
impl Forget for Param
impl Freeze for Param
impl Move for Param
impl RefUnwindSafe for Param
impl Send for Param
impl Sync for Param
impl Unpin for Param
impl UnsafeUnpin for Param
impl UnwindSafe for Param
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> DeserializeOwned for Twhere
T: for<'de> Deserialize<'de>,
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more