Expand description
§llm_devices: Device management and build system for LLM inference
The llm_devices crate is a workspace member of the llm_client project. It is used as a dependency by the llm_interface crate for building llama.cpp.
§Features
- Automated building of llama.cpp with appropriate platform-specific optimizations
- Device detection and configuration for CPU, RAM, CUDA (Linux/Windows), and Metal (macOS)
- Manages memory by detecting available VRAM/RAM, estimating model fit, and distributing layers across devices
- Logging tools
Structs§
- CpuConfig
- Configuration for managing CPU resources in LLM inference workloads.
- Cuda
Config - Configuration for NVIDIA CUDA devices on Linux and Windows platforms.
- Device
Config - Configuration for hardware devices used in LLM inference.
- Logging
Config - Configuration for the logging system.
Traits§
- Logging
Config Trait - Trait for configuring logging behavior.
Functions§
- build_
repo - Clones and builds a repository at a specified tag with appropriate platform-specific optimizations.
- get_
target_ directory - Resolves the Cargo target directory path.
- i_ln
- Writes an indented line without newline.
- i_lns
- Writes multiple indented lines without newlines.
- i_nln
- Writes an indented line with newline.
- i_nlns
- Writes multiple indented lines with newlines.
- init_
nvml_ wrapper - Initializes NVIDIA Management Library (NVML).