Struct Proxy 

pub struct Proxy<D: LlmDaemon + Debug> { /* private fields */ }

A proxy to actual LLM server, and only send the last pending request. If the LLM generation is slower than requests, then processing the oldest request does not make sense, as the user is more interested in the later requests. But current server (AFAIK) does not handle this properly…

Implementations

impl<D: LlmDaemon + Debug> Proxy<D>

pub fn new(config: ProxyConfig, inner: D) -> Self

Trait Implementations

impl<D: Debug + LlmDaemon + Debug> Debug for Proxy<D>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<D: LlmDaemon + Debug> LlmDaemon for Proxy<D>

type Config = ProxyConfig

fn fork_daemon(&self) -> Result<()>

Spawns the daemon, initializing any necessary resources or processes. This method is expected to be called before creation of tokio runtime, mostly due to the use of the fork. User is free to use async runtime after calling this.

fn heartbeat<'a, 'b>(&'b self) -> impl Future<Output = Result<()>> + Send + 'a

where 'a: 'b,

Creates a task which maintains a periodic heartbeat to the daemon. Daemon is expected to terminate if there’s no heartbeat for a certain period of time. Keeping this task within async runtime will ensure that the daemon is kept running during the application. FIXME: Extract heartbeat / ready to Async util?

fn config(&self) -> &Self::Config

fn ping(&self) -> Result<()>

Send a single heartbeat to let daemon know there is a client. Can be useful if you don’t want to use heartbeat.

fn ready<'a>(&self) -> impl Future<Output = ()> + Send + 'a

FIXME: Extract heartbeat / ready to Async util?