Struct GgmlContext 

pub struct GgmlContext { /* private fields */ }

A safe wrapper around ggml_context.

Implementations

impl GgmlContext

pub fn new(mem_size: usize, no_alloc: bool) -> Self

Create a new ggml context.

Parameters

• mem_size: Memory pool size in bytes for tensor metadata.
• no_alloc: If true, tensor data is not allocated (use with backend allocation).

Sizing the pool

Every tensor and graph created from this context is carved out of mem_size, and running out is not a recoverable error:

• a release build of ggml returns null, which the constructors here turn into a panic;
• a debug build of ggml calls GGML_ABORT and kills the process before returning, so no amount of Rust-side care helps.

So size it up front rather than guessing. Self::sized_for does the arithmetic; Self::used_mem and Self::mem_size let you check headroom while building a graph.

Panics

Panics if ggml returns a null pointer.

pub fn sized_for(n_tensors: usize, n_graphs: usize, no_alloc: bool) -> Self

Create a context sized for n_tensors tensors and n_graphs graphs.

Turns “pick a number and hope” into arithmetic: the pool holds metadata, whose per-item cost ggml reports through tensor_overhead and graph_overhead. A little slack is added for ggml’s own bookkeeping.

Views, reshapes and every intermediate an operation produces are tensors too — count graph nodes, not just the tensors you name.

// Room for 64 tensors and one graph.
let ctx = GgmlContext::sized_for(64, 1, true);
assert!(ctx.mem_size() >= 64 * llama_cpp_4::ggml::tensor_overhead());

Panics

Panics if ggml returns a null pointer.

pub fn used_mem(&self) -> usize

Bytes of the pool used so far.

Compare with Self::mem_size while building a graph to see whether it will fit, rather than finding out by aborting.

pub fn mem_size(&self) -> usize

Total size of the pool, as passed to Self::new.

pub fn free_mem(&self) -> usize

Bytes still available in the pool.

pub fn as_ptr(&self) -> *mut ggml_context

Get the raw context pointer.

pub fn new_tensor_1d(&self, typ: ggml_type, ne0: i64) -> GgmlTensor

Create a 1D tensor.

Panics

Panics if ggml returns null, which in a release build means the context’s memory pool is exhausted. A debug build of ggml aborts before returning — see GgmlContext::new for how to size the pool.

pub fn new_tensor_2d(&self, typ: ggml_type, ne0: i64, ne1: i64) -> GgmlTensor

Create a 2D tensor.

Panics

Panics if ggml returns null, which in a release build means the context’s memory pool is exhausted. A debug build of ggml aborts before returning — see GgmlContext::new for how to size the pool.

pub fn new_tensor_3d( &self, typ: ggml_type, ne0: i64, ne1: i64, ne2: i64, ) -> GgmlTensor

Create a 3D tensor.

Panics

Panics if ggml returns null, which in a release build means the context’s memory pool is exhausted. A debug build of ggml aborts before returning — see GgmlContext::new for how to size the pool.

pub fn new_tensor_4d( &self, typ: ggml_type, ne0: i64, ne1: i64, ne2: i64, ne3: i64, ) -> GgmlTensor

Create a 4D tensor.

Panics

Panics if ggml returns null, which in a release build means the context’s memory pool is exhausted. A debug build of ggml aborts before returning — see GgmlContext::new for how to size the pool.

pub fn dup_tensor(&self, src: &GgmlTensor) -> GgmlTensor

Create a tensor with the same shape and type as another.

Panics

Panics if ggml returns null, which in a release build means the context’s memory pool is exhausted. A debug build of ggml aborts before returning — see GgmlContext::new for how to size the pool.

pub fn new_tensor(&self, typ: ggml_type, ne: &[i64]) -> GgmlTensor

Create a new tensor with arbitrary dimensions.

Panics

Panics if ggml returns null, which in a release build means the context’s memory pool is exhausted. A debug build of ggml aborts before returning — see GgmlContext::new for how to size the pool.

pub fn add(&self, a: &GgmlTensor, b: &GgmlTensor) -> GgmlTensor

Element-wise addition: a + b

Panics

Panics if ggml returns null, which in a release build means the context’s memory pool is exhausted. A debug build of ggml aborts before returning — see GgmlContext::new for how to size the pool.

pub fn mul_mat(&self, a: &GgmlTensor, b: &GgmlTensor) -> GgmlTensor

Matrix multiplication: a @ b

Panics

Panics if ggml returns null, which in a release build means the context’s memory pool is exhausted. A debug build of ggml aborts before returning — see GgmlContext::new for how to size the pool.

pub fn scale(&self, a: &GgmlTensor, s: f32) -> GgmlTensor

Scale tensor: a * s

Panics

Panics if ggml returns null, which in a release build means the context’s memory pool is exhausted. A debug build of ggml aborts before returning — see GgmlContext::new for how to size the pool.

pub fn cast(&self, a: &GgmlTensor, typ: ggml_type) -> GgmlTensor

Cast tensor to a different type.

Panics

Panics if ggml returns null, which in a release build means the context’s memory pool is exhausted. A debug build of ggml aborts before returning — see GgmlContext::new for how to size the pool.

pub fn cont(&self, a: &GgmlTensor) -> GgmlTensor

Make tensor contiguous in memory.

Panics

Panics if ggml returns null, which in a release build means the context’s memory pool is exhausted. A debug build of ggml aborts before returning — see GgmlContext::new for how to size the pool.

pub fn transpose(&self, a: &GgmlTensor) -> GgmlTensor

Transpose a tensor.

Panics

Panics if ggml returns null, which in a release build means the context’s memory pool is exhausted. A debug build of ggml aborts before returning — see GgmlContext::new for how to size the pool.

pub fn reshape_1d(&self, a: &GgmlTensor, ne0: i64) -> GgmlTensor

Reshape to 1D.

Panics

Panics if ggml returns null, which in a release build means the context’s memory pool is exhausted. A debug build of ggml aborts before returning — see GgmlContext::new for how to size the pool.

pub fn reshape_2d(&self, a: &GgmlTensor, ne0: i64, ne1: i64) -> GgmlTensor

Reshape to 2D.

Panics

Panics if ggml returns null, which in a release build means the context’s memory pool is exhausted. A debug build of ggml aborts before returning — see GgmlContext::new for how to size the pool.

pub fn view_1d(&self, a: &GgmlTensor, ne0: i64, offset: usize) -> GgmlTensor

Create a 1D view of a tensor.

Panics

Panics if ggml returns null, which in a release build means the context’s memory pool is exhausted. A debug build of ggml aborts before returning — see GgmlContext::new for how to size the pool.

pub fn new_graph(&self) -> GgmlGraph

Create a new computation graph.

Panics

Panics if ggml returns null, which in a release build means the context’s memory pool is exhausted. A debug build of ggml aborts before returning — see GgmlContext::new for how to size the pool.

pub fn first_tensor(&self) -> Option<GgmlTensor>

Get the first tensor in this context.

pub fn next_tensor(&self, tensor: &GgmlTensor) -> Option<GgmlTensor>

Get the next tensor after tensor in this context.

Trait Implementations

impl Debug for GgmlContext

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Drop for GgmlContext

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.