Struct DeviceConfig 

pub struct DeviceConfig {
    pub cpu_config: CpuConfig,
    pub ram_config: RamConfig,
    pub use_gpu: bool,
    pub cuda_config: Option<CudaConfig>,
    pub error_on_config_issue: bool,
    pub layer_count: Option<u64>,
    pub average_layer_size_bytes: Option<u64>,
    pub local_model_path: String,
}

Configuration for device-specific settings in LLM inference.

Fields

cpu_config: CpuConfig

CPU configuration for thread count.

ram_config: RamConfig

RAM configuration for non-GPU inference on Windows and Unix.

This setting is used when GPU acceleration is not available or not enabled.

use_gpu: bool

Indicates whether to use any available GPUs for inference.

If true, the system will attempt to use GPU acceleration. If false, inference will be performed on CPU only.

cuda_config: Option<CudaConfig>

CUDA configuration for GPU inference on non-macOS platforms.

This field is only available on platforms other than macOS. If None, default CUDA settings will be used when GPU is enabled.

error_on_config_issue: bool

Determines error handling behavior for configuration issues.

If true, the system will return an error when encountering configuration issues. If false (default), issues will be logged and execution will continue if possible.

This flag is useful for debugging purposes.

layer_count: Option<u64>

The number of layers in the model.

This is set at runtime.

average_layer_size_bytes: Option<u64>

The average size of a layer in bytes.

This is set at runtime.

local_model_path: String

The file system path to the local model.

This is set at runtime.

Implementations

impl DeviceConfig

pub fn initialize(&mut self) -> Result<()>

pub fn available_memory_bytes(&self) -> Result<u64>

pub fn average_layer_size_bytes(&self) -> Result<u64>

pub fn layer_count(&self) -> Result<u64>

pub fn main_gpu(&self) -> Result<u32>

pub fn gpu_count(&self) -> usize

pub fn allocate_layers_to_gpus( &self, buffer_layer_per_gpu: u64, buffer_layer_main_gpu: u64, ) -> Result<Vec<GpuDevice>>

Trait Implementations

impl Clone for DeviceConfig

fn clone(&self) -> DeviceConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DeviceConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for DeviceConfig

fn default() -> Self

Returns the “default value” for a type.

impl Display for DeviceConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.