Struct LlmPool 

pub struct LlmPool { /* private fields */ }

Pool of LLM clients for different purposes.

This provides a centralized way to access LLM clients configured for specific tasks:

• Summary — Document summarization (fast, cheap model)
• Retrieval — Document navigation (capable model)
• TOC — Table of contents processing (fast, cheap model)

Example

use vectorless::llm::LlmPool;

let pool = LlmPool::from_defaults();

// Use summary client for summarization
let summary = pool.summary().complete(
    "You summarize text concisely.",
    "Long text to summarize..."
).await?;

// Use retrieval client for navigation
let nav = pool.retrieval().complete(
    "You navigate documents.",
    "Find information about X..."
).await?;

Implementations

impl LlmPool

pub fn new(configs: LlmConfigs) -> Self

Create a new LLM pool from configurations.

pub fn from_defaults() -> Self

Create a pool with default configurations.

Uses auto-detected models based on available API keys:

• OpenAI: gpt-4o-mini for summary/toc, gpt-4o for retrieval
• Anthropic: claude-3-haiku for summary/toc, claude-3-sonnet for retrieval
• Default: glm-4-flash for summary/toc, glm-4 for retrieval

pub fn with_concurrency(self, controller: ConcurrencyController) -> Self

Add concurrency control to all clients in the pool.

All clients share the same ConcurrencyController, which means rate limiting and concurrency limits are applied globally across all LLM operations.

Example

use vectorless::llm::LlmPool;
use vectorless::throttle::{ConcurrencyController, ConcurrencyConfig};

let config = ConcurrencyConfig::new()
    .with_max_concurrent_requests(10)
    .with_requests_per_minute(500);

let pool = LlmPool::from_defaults()
    .with_concurrency(ConcurrencyController::new(config));

pub fn with_shared_concurrency( self, controller: Arc<ConcurrencyController>, ) -> Self

Add concurrency control from an existing Arc.

pub fn concurrency(&self) -> Option<&ConcurrencyController>

Get the concurrency controller (if any).

pub fn summary(&self) -> &LlmClient

Get the summary client.

Used for generating summaries of document sections. Typically uses a fast, cost-effective model.

pub fn retrieval(&self) -> &LlmClient

Get the retrieval client.

Used for document navigation and retrieval. Typically uses a more capable model for better navigation decisions.

pub fn toc(&self) -> &LlmClient

Get the TOC client.

Used for TOC detection, parsing, and page assignment. Typically uses a fast, cost-effective model.

pub fn get(&self, purpose: &str) -> Option<&LlmClient>

Get a client for a specific purpose by name.

Arguments

• purpose - One of: “summary”, “summarize”, “retrieval”, “retrieve”, “navigate”, “toc”

Returns

Returns None if the purpose is not recognized.

pub fn single_model(model: impl Into<String>) -> Self

Create a pool with a single model for all purposes.

Useful for testing or simple deployments.

Trait Implementations

impl Clone for LlmPool

fn clone(&self) -> LlmPool

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for LlmPool

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for LlmPool

fn default() -> Self

Returns the “default value” for a type.