Struct RpcBackend 

pub struct RpcBackend { /* private fields */ }

RPC backend for distributed inference across multiple machines

Implementations

impl RpcBackend

pub fn init(endpoint: &str) -> Result<Self, RpcError>

Initialize a new RPC backend for the given endpoint

Arguments

• endpoint - The RPC server endpoint (e.g., “127.0.0.1:50052”)

Example

use llama_cpp_4::rpc::RpcBackend;

let backend = RpcBackend::init("127.0.0.1:50052")?;

pub fn is_rpc(&self) -> bool

Check if a backend is an RPC backend

pub fn buffer_type(&self) -> Option<NonNull<ggml_backend_buffer_type>>

Get the buffer type for this RPC backend

pub fn get_device_memory(&self) -> Result<(usize, usize), RpcError>

Query the available memory on the remote device

Returns (free_memory, total_memory) in bytes

pub fn endpoint(&self) -> &str

Get the endpoint this backend is connected to

Trait Implementations

impl Debug for RpcBackend

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Drop for RpcBackend

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.

impl Send for RpcBackend

impl Sync for RpcBackend