Struct RpcBackend 

pub struct RpcBackend { /* private fields */ }

RPC backend for distributed inference across multiple machines

Implementations

impl RpcBackend

pub fn init(endpoint: &str, device: u32) -> Result<Self, RpcError>

Initialize a new RPC backend for the given endpoint and remote device.

Arguments

• endpoint - The RPC server endpoint (e.g., “127.0.0.1:50052”)
• device - Index of the device to use on the remote server. A single endpoint can expose several devices; pass 0 for the first.

Errors

Returns RpcError::StringConversion if endpoint contains an interior NUL, or RpcError::InitializationFailed if llama.cpp could not reach the endpoint or the device index is out of range.

Example

use llama_cpp_4::rpc::RpcBackend;

let backend = RpcBackend::init("127.0.0.1:50052", 0)?;

pub fn is_rpc(&self) -> bool

Check if a backend is an RPC backend

pub fn buffer_type(&self) -> Option<NonNull<ggml_backend_buffer_type>>

Get the buffer type for this RPC backend

pub fn get_device_memory(&self) -> Result<(usize, usize), RpcError>

Query the available memory on the remote device

Returns (free_memory, total_memory) in bytes.

Errors

Returns RpcError::MemoryQueryFailed when the server reports a total of zero, which is how an unreachable endpoint surfaces here.

pub fn endpoint(&self) -> &str

Get the endpoint this backend is connected to

pub fn device(&self) -> u32

Index of the remote device this backend is bound to.

pub fn as_ptr(&self) -> NonNull<ggml_backend>

Get the raw backend pointer for FFI calls.

The pointer is owned by this RpcBackend and is freed on drop; do not free it, and do not use it after this value goes out of scope.

Trait Implementations

impl Debug for RpcBackend

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Drop for RpcBackend

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.

impl Send for RpcBackend

impl Sync for RpcBackend