Crate ggml

Expand description

ggml is a semi-idiomatic wrapper for the ggml C library.

It exposes a subset of operations (currently used to implement the llm library). Note that it does not expose a fully-idiomatic safe Rust interface; operations that could be potentially unsafe are marked as such.

ggml operates on a computational graph; no values will be computed until Context::graph_compute is executed. All Tensors are nodes in this computational graph, and values cannot be retrieved until computation is completed.

Modules§

format: Loading and saving of GGML files.
util: Utilities for reading and writing.

Structs§

Buffer: A buffer of memory that can be used as a scratch buffer for a Context.
ComputationGraph: A ggml computation graph. Keeps track of all state during computation.
Context: Acts as a RAII-guard over a sys::ggml_context, allocating via ggml_init and dropping via ggml_free.
QuantizationResult: Contains the result of a quantization operation.
Tensor: Tensors are owned by the context. A tensor is alive as long as the underlying context it was created with is alive.

Enums§

ContainerType: The format of the file containing the model.
Type: The type of a value in ggml.

Constants§

FILE_MAGIC_GGJT: Magic constant for ggml files (versioned, ggjt).
FILE_MAGIC_GGMF: Magic constant for ggml files (versioned, ggmf).
FILE_MAGIC_UNVERSIONED: Magic constant for ggml files (unversioned).
FORMAT_VERSION: The currently-supported format version for ggml files.
OBJECT_SIZE: The size of a ggml object.

Functions§

blck_size: The size of a block for t. Only relevant for quantized types.
quantize_q4_0: Quantizes src into dst using q4_0 quantization.
quantize_q4_1: Quantizes src into dst using q4_1 quantization.
type_size: The size of t as bytes.
type_sizef: type_size/blck_size as float.

Type Aliases§

ElementType: The type of a tensor element.