Struct FleetManager 

pub struct FleetManager { /* private fields */ }

Implementations

impl FleetManager

pub fn new( nodes: Vec<NodeSpec>, policy: PlacementPolicy, lease: Lease, ) -> FleetManager

pub fn with_pool_high( nodes: Vec<NodeSpec>, policy: PlacementPolicy, lease: Lease, pool_high_pct: u32, ) -> FleetManager

pub fn with_options( nodes: Vec<NodeSpec>, policy: PlacementPolicy, lease: Lease, pool_high_pct: u32, conns_per_node: usize, ) -> FleetManager

pub fn adopt( &self, head: FleetHead, peer: IpAddr, ) -> Result<(usize, bool, usize), String>

Takes in a node that dialed the head: one seen before, by identity, gets its slot back; a new one gets a slot of its own. true for a new one. Takes in a connection from a node that dialed the head. The node has a slot per lane it advertised and fills one per connection it opens, so it runs as many generations at once as it dialed; a node seen before, by identity, takes its slots back, unless it is still connected from another address, which is a second machine with the same name. Returns the node’s index, whether it is new, and its open connections.

pub fn connections(&self, i: usize) -> usize

The connections a node has open.

pub fn load(&self) -> Vec<usize>

pub fn status(&self) -> Vec<NodeStatus>

Every node the head knows, in the order it learned them.

pub fn nodes_up(&self) -> (usize, usize)

The nodes up now that serve model, and every node the head knows. Nodes up of all the head knows, whatever each serves.

pub fn serving(&self, model: &str) -> (usize, usize)

pub fn node_of(&self, session: u64) -> Option<String>

pub fn with_retry_every(self, every: Duration) -> FleetManager

How often a node that failed is tried again in the background (default RETRY_DOWN_AFTER).

pub fn expect_tokenizer(&self, model: &str, digest: u64)

Serves model only on nodes whose tokenizer has digest (crate::fleet::tokenizer_digest); a node that does not say is taken at its word.

pub fn with_liveness(self, liveness: Liveness) -> FleetManager

How the head’s connections are kept alive (default Liveness::DEFAULT); the background tend reads each idle one every heartbeat.

pub fn with_warm_log(self, path: PathBuf) -> FleetManager

Remembers in path which node ran each prompt prefix, so a restarted head still places a conversation’s next turn on the node that has it warm.

pub fn advertised_context(&self, model: &str) -> Option<u64>

pub fn place( &self, session: u64, params: GenParams, qos_class: u8, codec: &str, model: &str, tokens: &[u32], expected_out: u64, ) -> Result<usize, DaemonError>

pub fn place_by( &self, session: u64, params: GenParams, qos_class: u8, codec: &str, model: &str, tokens: &[u32], expected_out: u64, deadline: Option<Instant>, ) -> Result<Option<usize>, DaemonError>

As Self::place, for a generation that must start by deadline: none once it passes before the session is placed (every connection to a node serving the model held by another generation meanwhile), rather than placed for a generation already too late.

pub fn append(&self, session: u64, tokens: &[u32]) -> Result<(), DaemonError>

pub fn generate( &self, session: u64, max_new_tokens: u64, max_wall_ms: u64, ) -> Result<FleetGeneration, DaemonError>

pub fn generate_streaming( &self, session: u64, max_new_tokens: u64, max_wall_ms: u64, on_tokens: impl FnMut(&[u32]), ) -> Result<FleetGeneration, DaemonError>

pub fn generate_streaming_with( &self, session: u64, max_new_tokens: u64, max_wall_ms: u64, opts: Option<&Ask<'_>>, on_logprobs: &mut dyn FnMut(&[TokenLogprobMsg]), on_tokens: impl FnMut(&[u32]), ) -> Result<FleetGeneration, DaemonError>

As Self::generate_streaming, with what the generation asks beyond its sampling, sent to whichever node runs it, and on_logprobs told the log-probabilities it asked for.

pub fn generate_streaming_by( &self, session: u64, max_new_tokens: u64, max_wall_ms: u64, deadline: Option<Instant>, opts: Option<&Ask<'_>>, on_logprobs: &mut dyn FnMut(&[TokenLogprobMsg]), on_tokens: impl FnMut(&[u32]), ) -> Result<FleetGeneration, DaemonError>

As Self::generate_streaming_with, for a generation that must start by deadline: the wait for the session’s connection counts against it, one that passes meanwhile ends the generation unstarted (expired, no tokens), and the node is given only the time left.

pub fn state_sync( &self, model: &str, req: StateSyncReqMsg, blobs: &[Blob], ) -> Result<StateSyncOkMsg, DaemonError>

Runs a state operation on one node serving model, the next one tried when a node fails.

pub fn state_sync_streaming( &self, model: &str, req: StateSyncReqMsg, blobs: &[Blob], on_segment: &mut dyn FnMut(TranscribeSegmentMsg), ) -> Result<StateSyncOkMsg, DaemonError>

FleetManager::state_sync for a streamed transcription, each segment told as the node has it. Once one has been told, a node lost after it fails the call rather than starting over on another, which would tell those segments twice.

pub fn state_sync_all( &self, model: &str, req: StateSyncReqMsg, ) -> Result<StateSyncOkMsg, DaemonError>

Runs a state operation on every node up that serves model, as a LoRA must be on each node a request can land on; the answer is the last node’s.

pub fn remote_tokenizer( self: &Arc<Self>, model: &str, ) -> Result<RemoteTokenizer, DaemonError>

The tokenizer of a node serving model, for a head with no copy of it; encoding asks a node serving the model each time.

pub fn tokenize( &self, model: &str, text: &TokenizeMsg, ) -> Result<Option<Vec<u32>>, DaemonError>

Text a node serving model encodes.

pub fn cancel(&self, session: u64) -> Result<(), DaemonError>

pub fn park(&self, session: u64) -> Result<(), DaemonError>

pub fn resume(&self, session: u64) -> Result<usize, DaemonError>

pub fn finish(&self, session: u64)

Ends a session the caller is done with: the head drops its record, and the node the context it kept (now if its connection is free, else the next time it is).

pub fn probe(&self) -> Vec<Result<NodeReport, DaemonError>>

Connects to every node once, so the head learns each one (or why it cannot) at start.

pub fn keepalive(&self)

Reads every idle connection, which sends the node a heartbeat when one is due and answers its lease renewals; drops the ones a node closed or went silent on, and finishes deferred session ends. A connection in use is read by the call using it.

pub fn retry_down(&self)

Tries every node that failed and is due a retry, in parallel and outside the connection slots; one that answers is placed on again. Requests never wait on a down node.

pub fn tend(&self)

pub fn spawn_keepalive(self: &Arc<Self>)