Struct FleetManager 

pub struct FleetManager { /* private fields */ }

Implementations

impl FleetManager

pub fn new( nodes: Vec<NodeSpec>, policy: PlacementPolicy, lease: Lease, ) -> FleetManager

pub fn with_pool_high( nodes: Vec<NodeSpec>, policy: PlacementPolicy, lease: Lease, pool_high_pct: u32, ) -> FleetManager

pub fn with_options( nodes: Vec<NodeSpec>, policy: PlacementPolicy, lease: Lease, pool_high_pct: u32, conns_per_node: usize, ) -> FleetManager

pub fn adopt( &self, head: FleetHead, peer: IpAddr, ) -> Result<(usize, bool, usize), String>

Takes in a node that dialed the head: one seen before, by identity, gets its slot back; a new one gets a slot of its own. true for a new one. Takes in a connection from a node that dialed the head. The node has a slot per lane it advertised and fills one per connection it opens, so it runs as many generations at once as it dialed; a node seen before, by identity, takes its slots back, unless it is still connected from another address, which is a second machine with the same name. Returns the node’s index, whether it is new, and its open connections.

pub fn connections(&self, i: usize) -> usize

The connections a node has open.

pub fn load(&self) -> Vec<usize>

pub fn node_of(&self, session: u64) -> Option<String>

pub fn with_retry_every(self, every: Duration) -> FleetManager

How often a node that failed is tried again in the background (default RETRY_DOWN_AFTER).

pub fn advertised_context(&self, model: &str) -> Option<u64>

pub fn place( &self, session: u64, params: GenParams, qos_class: u8, codec: &str, model: &str, tokens: &[u32], expected_out: u64, ) -> Result<usize, DaemonError>

pub fn append(&self, session: u64, tokens: &[u32]) -> Result<(), DaemonError>

pub fn generate( &self, session: u64, max_new_tokens: u64, max_wall_ms: u64, ) -> Result<FleetGeneration, DaemonError>

pub fn generate_streaming( &self, session: u64, max_new_tokens: u64, max_wall_ms: u64, on_tokens: impl FnMut(&[u32]), ) -> Result<FleetGeneration, DaemonError>

pub fn cancel(&self, session: u64) -> Result<(), DaemonError>

pub fn park(&self, session: u64) -> Result<(), DaemonError>

pub fn resume(&self, session: u64) -> Result<usize, DaemonError>

pub fn finish(&self, session: u64)

Ends a session the caller is done with: the head drops its record, and the node the context it kept (now if its connection is free, else the next time it is).

pub fn probe(&self) -> Vec<Result<NodeReport, DaemonError>>

Connects to every node once, so the head learns each one (or why it cannot) at start.

pub fn keepalive(&self)

Keeps idle connections open, drops ones a node closed, and finishes deferred session ends.

pub fn retry_down(&self)

Tries every node that failed and is due a retry, in parallel and outside the connection slots; one that answers is placed on again. Requests never wait on a down node.

pub fn tend(&self)

pub fn spawn_keepalive(self: &Arc<Self>)