pub struct Lora {
pub down: Param,
pub up: Param,
pub scale: f32,
}Expand description
A low-rank adapter over a Linear: W + (alpha / rank) * up * down.
down is [rank, in_features] and up is [out_features, rank], so the
pair costs rank * (in + out) weights where the projection it adapts costs
in * out. up starts at zero, so attaching an adapter does not change
what the model predicts; the first optimizer step is what moves it.
Fields§
§down: Param§up: Param§scale: f32alpha / rank. Folded into the [tokens, rank] intermediate, which is
the smallest matrix in the adapter, so neither pass scales a full-width
one.
Implementations§
Source§impl Lora
impl Lora
Sourcepub fn new(
in_features: usize,
out_features: usize,
rank: usize,
alpha: f32,
rng: &mut StdRng,
) -> Self
pub fn new( in_features: usize, out_features: usize, rank: usize, alpha: f32, rng: &mut StdRng, ) -> Self
down is initialized like any other projection and up is zero, which
is the usual asymmetry: two random factors would start the adapter at a
random perturbation of a trained model.
pub fn rank(&self) -> usize
pub fn params_mut(&mut self) -> Vec<&mut Param>
Trait Implementations§
Source§impl<'de> Deserialize<'de> for Lora
impl<'de> Deserialize<'de> for Lora
Source§fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
__D: Deserializer<'de>,
fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
__D: Deserializer<'de>,
Deserialize this value from the given Serde deserializer. Read more
impl StructuralPartialEq for Lora
Auto Trait Implementations§
impl Forget for Lora
impl Freeze for Lora
impl Move for Lora
impl RefUnwindSafe for Lora
impl Send for Lora
impl Sync for Lora
impl Unpin for Lora
impl UnsafeUnpin for Lora
impl UnwindSafe for Lora
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> DeserializeOwned for Twhere
T: for<'de> Deserialize<'de>,
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more