Expand description
LLM provider connections for OxiFY
Structs§
- Alternative
Model - Alternative model suggestion
- Anthropic
Provider - Anthropic (Claude) provider implementation
- Batch
Config - Configuration for batch processing
- Batch
Provider - Batch provider that wraps any LLM provider with batching capabilities
- Batch
Stats - Statistics about batch processing
- Bedrock
Provider - AWS Bedrock provider for Claude models
- Budget
Limit - Budget limit configuration
- Budget
Provider - A provider that enforces budget limits
- Cache
Stats - Cache statistics
- Cached
Provider - A wrapper that adds caching functionality to any LLM provider
- Chain
OfThought - Chain-of-thought prompt builder
- Circuit
Breaker Config - Circuit breaker configuration
- Circuit
Breaker Provider - Circuit breaker wrapper for LLM providers
- Code
Block - Code block extracted from LLM response
- Cohere
Provider - Cohere provider
- Compression
Stats - Compression statistics
- Content
Length Interceptor - Interceptor that enforces maximum content length
- Contextual
Error - Enhanced error with context
- Dedup
Provider - Request deduplication provider
- Dedup
Stats - Deduplication statistics
- Embedding
Batch Provider - Batch provider for embeddings
- Embedding
Interceptor Provider - Provider wrapper for embedding interceptors
- Embedding
Request - Embedding
Response - Embedding
Usage - Error
Context - Error context for better debugging
- Error
Context Builder - Helper to create common error contexts
- Example
- Example for few-shot learning
- Fallback
Provider - A provider that automatically falls back to alternative providers on failure
- FewShot
Prompt - Few-shot prompt builder
- Gemini
Provider - Google Gemini provider
- Health
Check Config - Health check configuration
- Health
Check Provider - Health check provider wrapper
- Health
Stats - Health statistics
- Image
Input - Image input for vision models
- Instruction
Prompt - Instruction-based prompt builder
- Interceptor
Provider - Provider wrapper that applies interceptors to requests and responses
- Llama
CppProvider - llama.cpp server provider for local LLM inference
- LlmCache
- In-memory LLM response cache
- LlmChunk
- A chunk of streamed LLM response
- LlmRequest
- LlmRequest
Builder - Builder for constructing LLM requests easily
- LlmResponse
- Load
Balancer - Load balancer that distributes requests across multiple providers
- Load
Balancer Stats - Load balancer statistics
- Logging
Interceptor - Interceptor that logs requests
- Metrics
- Metrics collector for LLM operations
- Metrics
Provider - Provider wrapper with metrics collection
- Mistral
Provider - Mistral AI provider
- Model
Limits - Token limits for common models
- Model
Pricing - Cost per 1K tokens for different providers/models (in USD cents) These are approximate values and should be updated as pricing changes
- Model
Recommendation - Model recommendation response
- Model
Recommender - Model recommender
- Model
Utils - Model name utilities
- Observable
Provider - Provider wrapper with tracing and logging
- Ollama
Provider - Ollama (local model) provider implementation
- OpenAI
Provider - OpenAI provider implementation
- Otel
Embedding Provider - OpenTelemetry embedding provider wrapper
- Otel
Provider - OpenTelemetry provider wrapper
- Priority
Queue Config - Configuration for priority queue
- Priority
Queue Provider - Priority queue provider that wraps any LLM provider
- Priority
Queue Stats - Statistics about priority queue operations
- Prompt
Compressor - Prompt compressor for optimizing token usage
- Prompt
Template - A prompt template with variable substitution support
- Provider
Metadata - Provider metadata for selection
- Provider
Selector - Smart provider selector with automatic fallback
- Quick
Request - Quick helpers for common request patterns
- Rate
Limit Config - Rate limit configuration
- Rate
Limit Provider - Rate limiting provider wrapper
- Rate
Limit Stats - Rate limit statistics
- Recommendation
Request - Model recommendation request
- Request
Validator - Validates LLM requests before sending them to providers
- Response
Attributes - OpenTelemetry response attributes
- Response
Utils - Response post-processing utilities
- Retry
Config - Configuration for retry behavior
- Retry
Provider - A wrapper that adds retry functionality to any LLM provider
- Role
Prompt - Role-based prompt builder
- Sanitization
Interceptor - Interceptor that sanitizes prompts by removing sensitive patterns
- Selection
Criteria - Selection criteria for choosing an LLM provider
- Semantic
Cache - Semantic cache using embeddings for similarity matching
- Semantic
Cache Stats - Statistics for semantic cache performance
- Semantic
Cached Provider - Provider wrapper that adds semantic caching
- Similarity
Threshold - Similarity threshold for semantic cache matching (0.0 to 1.0)
- Span
Attributes - OpenTelemetry span attributes for LLM requests
- Stream
Usage - System
Prompts - Common system prompts for different personas
- Template
Library - Template library for common prompt patterns
- Timeout
Config - Configuration for timeout behavior
- Timeout
Provider - A wrapper that adds timeout functionality to any LLM provider
- Token
Utils - Token estimation utilities
- Tool
- Tool/Function definition for function calling
- Tool
Call - Tool/Function call made by the LLM
- Trace
Event - OpenTelemetry trace event
- Tracked
Provider - A wrapper that tracks usage for any LLM provider
- Usage
- Usage
Stats - Accumulated usage statistics
- Usage
Tracker - Thread-safe usage tracker
- Validation
Rules - Validation rules for LLM requests
- Vllm
Provider - vLLM provider for high-throughput LLM inference
- Workflow
Embedding Provider - Embedding provider wrapper that tracks costs per workflow
- Workflow
Provider - LLM provider wrapper that tracks costs per workflow
- Workflow
Stats - Statistics for a workflow
- Workflow
Tracker - Tracks costs and usage per workflow
Enums§
- Budget
Constraint - Budget constraint for model selection
- Circuit
State - Circuit breaker state
- Health
Status - Health status
- Image
Source Type - LlmError
- Load
Balancing Strategy - Load balancing strategy
- Optimization
Goal - Priority for model selection
- Request
Priority - Request priority levels
- Role
- Role in a conversation
- UseCase
- Use case categories for model selection
Traits§
- Embedding
Provider - Trait for embedding providers
- Embedding
Request Interceptor - Trait for intercepting embedding requests
- Embedding
Response Interceptor - Trait for intercepting embedding responses
- Error
Context Ext - Extension trait for adding context to errors
- LlmProvider
- Trait for LLM providers
- Request
Interceptor - Trait for intercepting and modifying requests before they are sent
- Response
Interceptor - Trait for intercepting and modifying responses after they are received
- Streaming
LlmProvider - Trait for LLM providers that support streaming