Struct RpcServer 

pub struct RpcServer { /* private fields */ }

RPC server for hosting a backend that can be accessed remotely

Implementations

impl RpcServer

pub fn start( backend: NonNull<ggml_backend>, endpoint: &str, free_mem: usize, total_mem: usize, ) -> Result<Self, RpcError>

Start an RPC server for the given backend

Arguments

• backend - The backend to expose via RPC
• endpoint - The endpoint to listen on (e.g., “0.0.0.0:50052”)
• free_mem - Amount of free memory to advertise (0 for auto)
• total_mem - Total memory to advertise (0 for auto)

Example

use llama_cpp_4::rpc::RpcServer;

// Assuming you have a backend initialized
let server = RpcServer::start(
    backend,
    "0.0.0.0:50052",
    0,
    0,
)?;

pub fn endpoint(&self) -> &str

Get the endpoint this server is listening on

pub fn backend(&self) -> NonNull<ggml_backend>

Get the backend this server is hosting

Trait Implementations

impl Debug for RpcServer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Send for RpcServer

impl Sync for RpcServer