pub struct GTensor<const DIMS: usize> { /* private fields */ }
Expand description
A GGML tensor. It uses a const generic for the dimensions.
Implementations§
Source§impl<const DIMS: usize> GTensor<DIMS>
impl<const DIMS: usize> GTensor<DIMS>
Sourcepub fn add<T: AsRef<GTensor<DIMS>>>(&self, rhs: T) -> Self
pub fn add<T: AsRef<GTensor<DIMS>>>(&self, rhs: T) -> Self
Add tensor B
to tensor A
.
Returns a new tensor.
a.add(b)
or a + b
Invariants
A
andB
must have the same shape.- Result will have the shape of
A
.
Example (pseudocode):
let a = [2, 2, 2];
let b = [1, 1, 1];
let result = a.add(b);
assert_eq!(result, [3, 3, 3]);
Sourcepub fn sub<T: AsRef<GTensor<DIMS>>>(&self, rhs: T) -> Self
pub fn sub<T: AsRef<GTensor<DIMS>>>(&self, rhs: T) -> Self
Subtract tensor B
from tensor A
.
Returns a new tensor.
a.sub(b)
or a - b
Invariants
A
andB
must have the same shape.- Result will have the shape of
A
.
Example (pseudocode):
let a = [3, 3, 3];
let b = [1, 1, 1];
let result = a.div(b);
assert_eq!(result, [2, 2, 2]);
Sourcepub fn mul<T: AsRef<GTensor<DIMS>>>(&self, rhs: T) -> Self
pub fn mul<T: AsRef<GTensor<DIMS>>>(&self, rhs: T) -> Self
Multiply tensor A
by tensor B
.
Returns a new tensor.
Note: This is elementwise multiplication, not matrix multiplication.
a.mul(b)
or a * b
Invariants
A
andB
must have the same shape.- Result will have the shape of
A
.
Example (pseudocode):
let a = [3, 3, 3];
let b = [2, 2, 2];
let result = a.mul(b);
assert_eq!(result, [6, 6, 6]);
Sourcepub fn div<T: AsRef<GTensor<DIMS>>>(&self, rhs: T) -> Self
pub fn div<T: AsRef<GTensor<DIMS>>>(&self, rhs: T) -> Self
Elementwise divide tensor A
by tensor B
.
Returns a new tensor.
a.div(b)
or a / b
Invariants
A
andB
must have the same shape.- Result will have the shape of
A
.
Example (pseudocode):
let a = [6, 6, 6];
let b = [2, 2, 2];
let result = a.div(b);
assert_eq!(result, [3, 3, 3]);
Sourcepub fn scale<const RDIMS: usize, T: AsRef<GTensor<RDIMS>>>(
&self,
rhs: T,
) -> Self
pub fn scale<const RDIMS: usize, T: AsRef<GTensor<RDIMS>>>( &self, rhs: T, ) -> Self
Scale tensor A
by tensor B
.
This is basically just scalar multiplication.
Returns a new tensor.
a.scale(b)
Invariants
- Tensor
B
must have shape[1]
. (AKA 1d tensor with a single item.) - Result will have the shape of
A
.
Example (pseudocode):
let a = [3, 3, 3];
let b = [2];
let result = a.scale(b);
assert_eq!(result, [6, 6, 6]);
Sourcepub fn repeat<const RDIMS: usize, T: AsRef<GTensor<RDIMS>>>(
&self,
rhs: T,
) -> GTensor<RDIMS>
pub fn repeat<const RDIMS: usize, T: AsRef<GTensor<RDIMS>>>( &self, rhs: T, ) -> GTensor<RDIMS>
Repeat tensor A
based on the shape of tensor B
.
The content of B
is not used.
Returns a new tensor.
a.repeat(b)
Invariants
- Both
A
andB
must be 1d or 2d tensors. - Neither
A
orB
can be transposed or permuted. - The shape of
B
must be divisible by the shape ofA
. In other words,b_rows % a_rows
andb_cols % a_cols
must both be0
. - Result will have the shape of
B
.
Example (pseudocode):
let a =
[ [2, 3],
[4, 5] ];
let b =
[ [1, 1, 1, 1],
[1, 1, 1, 1],
[1, 1, 1, 1],
[1, 1, 1, 1] ];
let expected =
[ [2, 3, 2, 3],
[4, 5, 4, 5],
[2, 3, 2, 3],
[4, 5, 4, 5] ];
let result = a.repeat(b);
assert_eq!(result, expected);
Sourcepub fn conv_1d<const RDIMS: usize, const ODIMS: usize, T: AsRef<GTensor<RDIMS>>>(
&self,
rhs: T,
s0: usize,
p0: usize,
d0: usize,
) -> Self
pub fn conv_1d<const RDIMS: usize, const ODIMS: usize, T: AsRef<GTensor<RDIMS>>>( &self, rhs: T, s0: usize, p0: usize, d0: usize, ) -> Self
Sourcepub fn permute(&self, axes: [usize; 4]) -> Self
pub fn permute(&self, axes: [usize; 4]) -> Self
Create a view A
of based on the specified order of dimensions.
Returns a new tensor.
Note: Dimensions start from 0
, so 0
is the 1st dimension, 3
is the 4th.
a.permute([4, 2, 1, 1])
Invariants
- The axes must be unique.
[0, 0, 1, 2]
would be invalid, for example. - The axes must be a valid dimension. In other words, a number between
0
and3
inclusive.
Example (pseudocode):
let a =
[ [1, 1, 1],
[2, 2, 3],
[3, 3, 3] ];
let expected =
[ [1, 2, 3],
[1, 2, 3],
[1, 2, 3] ];
let result = a.permute([1, 0, 2, 3]);
assert_eq!(result, expected);
Source§impl<const DIMS: usize> GTensor<DIMS>
impl<const DIMS: usize> GTensor<DIMS>
Sourcepub fn map_unary(
&self,
fun: unsafe extern "C" fn(arg1: c_int, arg2: *mut f32, arg3: *const f32),
) -> Self
pub fn map_unary( &self, fun: unsafe extern "C" fn(arg1: c_int, arg2: *mut f32, arg3: *const f32), ) -> Self
Elementwise unary map operation on tensor A
.
Returns a new tensor.
Note: This function is rather unfriendly to use directly.
See the map_unop! macro which will help you create the required
unsafe extern "C"
function.
a.map_unary(unary_fun_ptr)
Invariants
- The tensor must be of type GType::F32.
- The result will be the same shape and type as
A
.
Example (pseudocode):
use std::ffi::c_int;
unsafe extern "C" fn unary_map_fn(n: c_int, dst: *mut f32, src: *const f32) {
let dst = ::std::slice::from_raw_parts_mut(dst, n as usize);
let src = ::std::slice::from_raw_parts(src, n as usize);
dst.iter_mut()
.zip(src.iter().copied())
.for_each(|(dst, src0)| *dst = src + 10);
}
let a = [1, 2, 3, 4];
let result = a.map_unary(unary_map_fn);
assert_eq!(result, [11, 12, 13, 14]);
Sourcepub fn map_binary<T: AsRef<GTensor<DIMS>>>(
&self,
rhs: T,
fun: unsafe extern "C" fn(arg1: c_int, arg2: *mut f32, arg3: *const f32, arg4: *const f32),
) -> Self
pub fn map_binary<T: AsRef<GTensor<DIMS>>>( &self, rhs: T, fun: unsafe extern "C" fn(arg1: c_int, arg2: *mut f32, arg3: *const f32, arg4: *const f32), ) -> Self
Elementwise binary map operation on tensors A
and B
.
Returns a new tensor.
Note: This function is rather unfriendly to use directly.
See the map_binop! macro which will help you create the required
unsafe extern "C"
function.
a.map_binary(b, binary_fun_ptr)
Invariants
- The tensor must be of type GType::F32.
A
andB
must be the same shape.- The result will be the same shape and type as
A
.
Example (pseudocode):
use std::ffi::c_int;
unsafe extern "C" fn binary_map_fn(
n: c_int,
dst: *mut f32,
src0: *const f32
src1: *const f32
) {
let dst = ::std::slice::from_raw_parts_mut(dst, n as usize);
let src0 = ::std::slice::from_raw_parts(src, n as usize);
let src1 = ::std::slice::from_raw_parts(src, n as usize);
dst.iter_mut()
.zip(src0.iter().copied())
.zip(src1.iter().copied())
.for_each(|((dst, src0), src1)| *dst = src0 + src1);
}
let a = [1, 2, 3, 4];
let a = [10, 10, 10, 10];
let result = a.map_binary(b, binary_map_fn);
assert_eq!(result, [11, 12, 13, 14]);
Source§impl<const DIMS: usize> GTensor<DIMS>
impl<const DIMS: usize> GTensor<DIMS>
Sourcepub const DIMS: usize = DIMS
pub const DIMS: usize = DIMS
Alternate method to access a tensor’s dimensions without needing an actual GTensor value to work with.
Sourcepub fn element_size(&self) -> usize
pub fn element_size(&self) -> usize
Returns the number of bytes each element uses.
Note: May not be accurate for quantized types.
Sourcepub fn shape(&self) -> [usize; DIMS]
pub fn shape(&self) -> [usize; DIMS]
Return the shape of this tensor as an array.
Note: The length of the shape array will be equal to the tensor’s dimensions.
Sourcepub fn ggml_op(&self) -> ggml_op
pub fn ggml_op(&self) -> ggml_op
Returns the GGML operation associated with this tensor if available.
Sourcepub fn element_type(&self) -> GType
pub fn element_type(&self) -> GType
Returns the element type.
Sourcepub fn metadata(&self) -> GTensorMetadata<DIMS>
pub fn metadata(&self) -> GTensorMetadata<DIMS>
Returns a copy of the metadata associated with this tensor.
Sourcepub fn get_ne(&self) -> [u32; 4]
pub fn get_ne(&self) -> [u32; 4]
Returns GGML’s conception of this tensor’s shape.
Note: This is a low level function. Be aware that GGML shapes have the first two dimensions swapped.
Sourcepub fn get_nb(&self) -> [u32; 4]
pub fn get_nb(&self) -> [u32; 4]
Returns GGML’s conception of this tensor’s strides in bytes.
Note: This is a low level function. Be aware that GGML shapes have the first two dimensions swapped. This also applies to the order of strides.
Note 2: Also be aware that the strides are based on bytes, and not the number of elements.
Sourcepub fn fill_i32(&mut self, val: i32)
pub fn fill_i32(&mut self, val: i32)
Immediately fills the tensor’s data with the specified i32
value.
Invariants
- The tensor’s type must not be quantized.
Sourcepub fn fill_f32(&mut self, val: f32)
pub fn fill_f32(&mut self, val: f32)
Immediately fills the tensor’s data with the specified f32
value.
Invariants
- The tensor’s type must not be quantized.
Sourcepub fn get_f32_1d(&self, index: usize) -> Result<f32>
pub fn get_f32_1d(&self, index: usize) -> Result<f32>
Immediately returns the value of an element at the
specified index as a f32
.
Invariants
- The tensor’s type must not be quantized.
- The index must be valid.
Sourcepub fn get_i32_1d(&self, index: usize) -> Result<i32>
pub fn get_i32_1d(&self, index: usize) -> Result<i32>
Immediately returns the value of an element at the
specified index as an i32
.
Invariants
- The tensor’s type must not be quantized.
- The index must be valid.
Sourcepub fn set_f32_1d(&mut self, index: usize, val: f32)
pub fn set_f32_1d(&mut self, index: usize, val: f32)
Immediately set the value of an element at the
specified index to the specified f32
value.
Invariants
- The tensor’s type must not be quantized.
- The index must be valid.
Sourcepub fn set_i32_1d(&mut self, index: usize, val: i32)
pub fn set_i32_1d(&mut self, index: usize, val: i32)
Immediately set the value of an element at the
specified index to the specified i32
value.
Invariants
- The tensor’s type must not be quantized.
- The index must be valid.
Source§impl<const DIMS: usize> GTensor<DIMS>
impl<const DIMS: usize> GTensor<DIMS>
Sourcepub unsafe fn with_data_mut<F, O>(&mut self, fun: F) -> Result<O>
pub unsafe fn with_data_mut<F, O>(&mut self, fun: F) -> Result<O>
Low level function that allows mutably accessing a tensor’s
data as a slice of u8
.
§Safety
Since this is working with the raw bytes, you need to be careful not to reinterpret as the wrong type or set the data to something that would contain an invalid value for the type.
Sourcepub unsafe fn with_data<F, O>(&self, fun: F) -> Result<O>
pub unsafe fn with_data<F, O>(&self, fun: F) -> Result<O>
Low level function that allows accessing a tensor’s
data as a slice of u8
.
§Safety
Since this is working with the raw bytes, you need to be careful not to reinterpret as the wrong type.
Sourcepub unsafe fn populate_raw<S: AsRef<[u8]>>(&mut self, data: S)
pub unsafe fn populate_raw<S: AsRef<[u8]>>(&mut self, data: S)
§Safety
Fills a tensor with raw data. It’s your responsibility to make sure the format is correct.
Source§impl<const DIMS: usize> GTensor<DIMS>
impl<const DIMS: usize> GTensor<DIMS>
Sourcepub fn copy_from<T: AsRef<GTensor<DIMS>>>(&mut self, rhs: T)
pub fn copy_from<T: AsRef<GTensor<DIMS>>>(&mut self, rhs: T)
Copies data from the specified tensor into this tensor when the graph runs.
Note: This immediately overwrites self
with the copy.
Sourcepub fn populate_f32<S: AsRef<[f32]>>(&mut self, data: S)
pub fn populate_f32<S: AsRef<[f32]>>(&mut self, data: S)
Immediately copy the specified f32
values into this tensor.
Invariants
- The tensor must be of type GType::F32.
- The length of the incoming data must match the size of the tensor.
Sourcepub fn copy_to_slice_f32<S: AsMut<[f32]>>(&self, dest: S) -> Result<()>
pub fn copy_to_slice_f32<S: AsMut<[f32]>>(&self, dest: S) -> Result<()>
Immediately copy the data from this tensor to the specified destination.
Invariants
- The tensor must be of type GType::F32.
- The length of the destination must match the size of the tensor.
- The destination must be elements of
f32
.
Source§impl<const DIMS: usize> GTensor<DIMS>
impl<const DIMS: usize> GTensor<DIMS>
Sourcepub fn sqr(&self) -> Self
pub fn sqr(&self) -> Self
Elementwise square of tensor A
.
This is the same as a map where each element is
multiplied by itself.
Returns a new tensor.
a.sqr()
Invariants
- Result will have the shape of
A
.
Example (pseudocode):
let a = [2, 2, 2];
let result = a.sqr();
assert_eq!(result, [4, 4, 4]);
Sourcepub fn sqrt(&self) -> Self
pub fn sqrt(&self) -> Self
Elementwise square root of tensor A
.
Returns a new tensor.
a.sqrt()
Invariants
- Result will have the shape of
A
.
Example (pseudocode):
let a = [9, 9, 9];
let result = a.sqrt();
assert_eq!(result, [3, 3, 3]);
Sourcepub fn abs(&self) -> Self
pub fn abs(&self) -> Self
Elementwise abs
of tensor A
.
Returns a new tensor.
a.abs()
Invariants
- Result will have the shape of
A
.
Example (pseudocode):
let a = [-1, -2, -3];
let result = a.abs();
assert_eq!(result, [1, 2, 3]);
Sourcepub fn sgn(&self) -> Self
pub fn sgn(&self) -> Self
Elementwise sign operation on tensor A
.
Returns a new tensor.
a.sgn()
Invariants
- If an element is 0 then the result will be 0.
- If an element is over 0 then the result will be 1.
- If an element is less than 0 then the result will be -1.
- Result will have the shape of
A
.
Example (pseudocode):
let a = [-5, 0, 6];
let result = a.sgn();
assert_eq!(result, [-1, 0, 1]);
Sourcepub fn neg(&self) -> Self
pub fn neg(&self) -> Self
Elementwise negation operation on tensor A
.
In other words, it just flips the sign.
Returns a new tensor.
a.neg()
Invariants
- Result will have the shape of
A
.
Example (pseudocode):
let a = [1, -1, -6, 7];
let result = a.sgn();
assert_eq!(result, [-1, 1, 6, -7]);
Sourcepub fn step(&self) -> Self
pub fn step(&self) -> Self
Elementwise step operation on tensor A
.
Returns a new tensor.
a.step()
See https://en.wikipedia.org/wiki/Activation_function
Invariants
- If an element is over 0 then the result will be 1.
- If an element is less or equal to 0 then the result will be 0.
- Result will have the shape of
A
.
Example (pseudocode):
let a = [1, -1, -6, 7];
let result = a.step();
assert_eq!(result, [1, 0, 0, 1]);
Sourcepub fn relu(&self) -> Self
pub fn relu(&self) -> Self
Perform ReLU operation on tensor A
.
Returns a new tensor.
a.relu()
See https://en.wikipedia.org/wiki/Activation_function
Invariants
- If an element is over 0 then the element passes through unchanged.
- If an element is less or equal to 0 then the result will be 0.
- Result will have the shape of
A
.
Example (pseudocode):
let a = [1, -1, -6, 7];
let result = a.relu();
assert_eq!(result, [1, 0, 0, 7]);
Sourcepub fn gelu(&self) -> Self
pub fn gelu(&self) -> Self
Perform GELU (AKA “Gaussian Error Linear Unit”)
operation on tensor A
.
Returns a new tensor.
a.gelu()
Sourcepub fn silu(&self) -> Self
pub fn silu(&self) -> Self
Perform SiLU (AKA “Sigmoid Linear Unit”)
operation on tensor A
.
Returns a new tensor.
a.silu()
Sourcepub fn transpose(&self) -> Self
pub fn transpose(&self) -> Self
Create a view A
with the first and second dimensions flipped.
Returns a new tensor.
Note: This is the same as
a.permute([1, 0, 2, 3])
.
a.transpose()
Example (pseudocode):
let a =
[ [1, 1, 1],
[2, 2, 3],
[3, 3, 3] ];
let expected =
[ [1, 2, 3],
[1, 2, 3],
[1, 2, 3] ];
let result = a.transpose();
assert_eq!(result, expected);
Sourcepub fn soft_max(&self) -> Self
pub fn soft_max(&self) -> Self
Apply the softmax
(AKA softargmax
or “normalized
exponential function”) to A
.
Returns a new tensor.
a.soft_max()
This one is a bit too complicated to explain here. See https://en.wikipedia.org/wiki/Softmax_function
Invariants
- Result will have the shape and type of
A
.
Sourcepub fn norm(&self, eps: f32) -> Self
pub fn norm(&self, eps: f32) -> Self
Perform LayerNorm operation on tensor A
.
Returns a new tensor.
a.norm()
See this helpful explanation for more information and comparison with the GTensor::rms_norm function.
Sourcepub fn rms_norm(&self, eps: f32) -> Self
pub fn rms_norm(&self, eps: f32) -> Self
Perform RMSNorm operation on tensor A
.
Returns a new tensor.
a.rms_norm()
See this helpful explanation for more information and comparison with the GTensor::norm function.
Sourcepub fn mean<const ODIMS: usize>(&self) -> GTensor<ODIMS>
pub fn mean<const ODIMS: usize>(&self) -> GTensor<ODIMS>
Elementwise mean
of tensor A
.
Returns a new tensor.
a.mean()
Invariants
- Result will be a 1 dimensional tensor with one item.
- The result tensor will have type GType::F32.
Example (pseudocode):
let a = [1, 2, 3, 4];
let result = a.mean();
assert_eq!(result, [2.5]);
Sourcepub fn sum<const ODIMS: usize>(&self) -> GTensor<ODIMS>
pub fn sum<const ODIMS: usize>(&self) -> GTensor<ODIMS>
Elementwise sum
of tensor A
.
Returns a new tensor.
a.sum()
Invariants
- Result will be a 1 dimensional tensor with one item.
- The result tensor will the same type as
A
.
Example (pseudocode):
let a = [1, 2, 3, 4];
let result = a.sum();
assert_eq!(result, [10]);