Struct DecodePeer 

pub struct DecodePeer { /* private fields */ }

A Decode rank’s PD transfer peer.

Implementations

impl DecodePeer

pub fn start(config: DecodeConfig) -> Result<(Self, DecodeOutbox), StartError>

Start the transfer engine, register the cache, open the status socket and start the threads.

pub fn port(&self) -> u16

The status socket’s port.

pub fn session_id(&self) -> String

The transfer engine’s session id.

Trait Implementations

impl Drop for DecodePeer

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.

impl TransferPeer for DecodePeer

fn abort_room(&mut self, _: &str, _: u64)

The decode node has no encoder to wait on.

fn register( &mut self, rid: &str, bootstrap: &BootstrapInfo, ) -> Result<(), PdError>

Track an admitted request, before the scheduler can plan it.

fn poll( &mut self, mesh: &mut Mesh, replica: &Group, ) -> Result<Vec<PdEvent>, PdError>

The transfers whose state changed since the last poll, as every member of replica sees them: a rank reports the least advanced state any member holds. Every member calls this in the same rounds.

fn record_usage( &mut self, _rid: &str, _cached_tokens: i64, _bootstrap_logprob: Option<f64>, )

P role: a committed prompt’s usage, sent with its completion: the prompt tokens served from the cache and the first token’s logprob.

fn has_in_flight(&self) -> bool

A transfer is still tracked: it may yet read or write the cache.