pub struct Q50Weight<R: Runtime> { /* private fields */ }Expand description
A weight matrix resident in VRAM in packed Q5_0 form ([n_out, k],
k % 32 == 0, blocks along k).
Implementations§
Source§impl<R: Runtime> Q50Weight<R>
impl<R: Runtime> Q50Weight<R>
Sourcepub fn from_gguf_bytes(
client: &ComputeClient<R>,
data: &[u8],
n_out: usize,
k: usize,
) -> Result<Self>
pub fn from_gguf_bytes( client: &ComputeClient<R>, data: &[u8], n_out: usize, k: usize, ) -> Result<Self>
Repacks a GGUF Q5_0 tensor onto the device.
Sourcepub fn vram_bytes(&self) -> usize
pub fn vram_bytes(&self) -> usize
Bytes in VRAM: 24 per 32 weights (6.0 bits/weight).
Sourcepub fn matmul_device(
&self,
client: &ComputeClient<R>,
x: Handle,
m: usize,
) -> Handle
pub fn matmul_device( &self, client: &ComputeClient<R>, x: Handle, m: usize, ) -> Handle
Device path: launch only, output handle returned.
Sourcepub fn matmul_host(
&self,
client: &ComputeClient<R>,
x: &[f32],
m: usize,
) -> Result<Vec<f32>>
pub fn matmul_host( &self, client: &ComputeClient<R>, x: &[f32], m: usize, ) -> Result<Vec<f32>>
Host-slice convenience for tests.
Auto Trait Implementations§
impl<R> !RefUnwindSafe for Q50Weight<R>
impl<R> !UnwindSafe for Q50Weight<R>
impl<R> Freeze for Q50Weight<R>
impl<R> Send for Q50Weight<R>
impl<R> Sync for Q50Weight<R>
impl<R> Unpin for Q50Weight<R>where
R: Unpin,
impl<R> UnsafeUnpin for Q50Weight<R>
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more