pub struct SwiGLU { /* private fields */ }Expand description
SwiGLU activation: Swish-Gated Linear Unit
A gated activation function that combines SiLU (Swish) with gating. Formula: down_proj(silu(gate) * up)
Implementations§
Source§impl SwiGLU
impl SwiGLU
Sourcepub fn new(
hidden_size: usize,
expansion: f32,
vb: VarBuilder<'_>,
) -> Result<Self>
pub fn new( hidden_size: usize, expansion: f32, vb: VarBuilder<'_>, ) -> Result<Self>
Create new SwiGLU layer
§Arguments
hidden_size- Input/output dimensionexpansion- Expansion factor for intermediate dimensionvb- VarBuilder for parameter initialization
pub fn forward(&self, x: &Tensor) -> Result<Tensor>
Auto Trait Implementations§
impl Freeze for SwiGLU
impl !RefUnwindSafe for SwiGLU
impl Send for SwiGLU
impl Sync for SwiGLU
impl Unpin for SwiGLU
impl !UnwindSafe for SwiGLU
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
Source§impl<T> Instrument for T
impl<T> Instrument for T
Source§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
Source§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more