Struct PrefillPeer 

pub struct PrefillPeer { /* private fields */ }

A Prefill rank’s PD transfer peer.

Implementations

impl PrefillPeer

pub fn start(config: PrefillConfig) -> Result<(Self, PrefillOutbox), StartError>

Start the transfer engine, register the cache, open the control socket, start the threads and register with the bootstrap server.

pub fn port(&self) -> u16

The control socket’s port.

pub fn embedding_receiver( &self, config: ReceiverConfig, ) -> Result<Receiver, StartError>

EPD, the first stage of a multimodal Prefill node: a receiver of the encode nodes’ embeddings, over this peer’s transfer engine.

pub fn session_id(&self) -> String

The transfer engine’s session id.

Trait Implementations

impl Drop for PrefillPeer

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.

impl TransferPeer for PrefillPeer

fn register( &mut self, rid: &str, bootstrap: &BootstrapInfo, ) -> Result<(), PdError>

Track an admitted request, before the scheduler can plan it.

fn poll( &mut self, mesh: &mut Mesh, replica: &Group, ) -> Result<Vec<PdEvent>, PdError>

The transfers whose state changed since the last poll, as every member of replica sees them: a rank reports the least advanced state any member holds. Every member calls this in the same rounds.

fn has_in_flight(&self) -> bool

A transfer is still tracked: it may yet read or write the cache.

fn abort_room(&mut self, rid: &str, room: u64)

EPD, P role: tell the decode node waiting on room room that the request failed before it could register (its embeddings never arrived), so its receiver fails instead of waiting. Best effort.

fn record_usage( &mut self, rid: &str, cached_tokens: i64, bootstrap_logprob: Option<f64>, )

P role: a committed prompt’s usage, sent with its completion: the prompt tokens served from the cache and the first token’s logprob.