flyllm 0.4.1

A Rust library for unifying LLM backends as an abstraction layer with load balancing.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

/// Load balancer module for distributing requests across multiple LLM providers
///
/// This module contains components for:
/// - Managing provider instances with metrics tracking
/// - Selecting appropriate providers based on tasks and load
/// - Implementing different load balancing strategies
/// - Handling retries and fallbacks when providers fail
/// - Tracking token usage across providers

pub mod tracker;
pub mod manager;
pub mod strategies;
pub mod tasks;
pub mod builder;
pub mod utils;
pub mod types;

pub use types::{GenerationRequest, LlmManagerResponse};
pub use manager::{LlmManager};
pub use tasks::TaskDefinition;