Crate llm_base

This crate provides a unified interface for loading and using Large Language Models (LLMs).

This is the base crate that implementors can use to implement their own LLMs.

As a user, you probably want to use the llm crate instead.

Re-exports

• pub use ggml;
• pub use model::Hyperparameters;
• pub use model::KnownModel;
• pub use model::Model;
• pub use model::ModelParameters;
• pub use model::OutputRequest;
• pub use util::TokenUtf8Buffer;

Modules

• model
  Large language model traits and types
• util
  Utilities for interacting with LLMs and loading them.

Structs

• InferenceParameters
  The parameters for text generation.
• InferenceRequest
  Settings specific to InferenceSession::infer.
• InferenceSession
  An inference session represents the state of the text generation. This holds the full context window, as well as several additional parameters used during sampling.
• InferenceSessionConfig
  Configuration for an inference session.
• InferenceSnapshot
  A serializable snapshot of the inference process. Can be restored by calling InferenceSession::from_snapshot.
• InferenceStats
  Statistics about the inference process.
• Loader
  A GGML format loader for LLMs.
• Mmap
  A handle to an immutable memory mapped buffer.
• TokenBias
  A list of tokens to bias during the process of inferencing.
• Vocabulary
  The vocabulary used by a model.

Enums

• ContainerType
  The format of the file containing the model.
• ElementType
  The type of a value in ggml.
• FileType
  How the tensors are stored in GGML LLM models.
• InferenceError
  Errors encountered during the inference process.
• LoadError
  Errors encountered during the loading process.
• LoadProgress
  Each variant represents a step within the process of loading the model. These can be used to report progress to the user.
• ModelKVMemoryType
  Allowed types for the model memory K/V tensors.
• QuantizeError
  Errors encountered during the quantization process.
• QuantizeProgress
  Progress of quantization.
• SnapshotError
  Errors encountered during the snapshot process.

Traits

• TensorLoader
  Used by models to fetch tensors from a loader.

Functions

• load
  Load a GGML model from the path and configure it per the params. The status of the loading process will be reported through load_progress_callback.
• load_progress_callback_stdout
  A implementation for load_progress_callback that outputs to stdout.
• quantize
  Quantizes a model.

Type Definitions

• TokenId
  The identifier of a token in a vocabulary.