Expand description
§Ferrum Server
HTTP API server abstractions for LLM inference services.
§Overview
This module defines the core traits for implementing HTTP API servers that can serve LLM inference requests with OpenAI API compatibility.
§Design Principles
- Framework Agnostic: Abstract interfaces that work with any HTTP framework
- OpenAI Compatible: Support for OpenAI Chat Completions API
- Middleware Support: Pluggable middleware for auth, logging, rate limiting
- Streaming Support: Server-Sent Events for streaming responses
- Monitoring: Built-in metrics and health checks
Re-exports§
pub use traits::AuthProvider;pub use traits::HttpServer;pub use traits::MiddlewareStack;pub use traits::RateLimiter as ServerRateLimiter;pub use traits::RequestHandler;pub use traits::ResponseBuilder;pub use traits::StreamingHandler;pub use types::ApiVersion;pub use types::Headers;pub use types::HealthStatus;pub use types::HttpMethod;pub use types::HttpRequest;pub use types::HttpResponse;pub use types::RequestContext;pub use types::ServerConfig;pub use types::ServerMetrics;pub use types::StatusCode;pub use openai::ChatCompletionsRequest;pub use openai::ChatCompletionsResponse;pub use openai::ChatFunction;pub use openai::ChatFunctionCall;pub use openai::ChatMessage;pub use openai::ChatTool;pub use openai::ChatToolCall;pub use openai::CompletionPrompt;pub use openai::CompletionsRequest;pub use openai::CompletionsResponse;pub use openai::FunctionCallChoice;pub use openai::ModelListResponse;pub use openai::OpenAiError;pub use openai::OpenAiErrorType;pub use openai::OpenAiResponseFormat;pub use openai::StreamOptions;pub use openai::ToolChoice;pub use middleware::AuthConfig;pub use middleware::CompressionConfig;pub use middleware::CorsConfig;pub use middleware::LoggingConfig;pub use middleware::MiddlewareConfig;pub use middleware::RateLimitConfig as MiddlewareRateLimitConfig;pub use axum_server::default_chat_sampling_params;pub use axum_server::init_prometheus_recorder;pub use axum_server::AxumServer;pub use model_registry::LoraAdapterModel;pub use model_registry::ServedModelEntry;pub use model_registry::ServedModelKind;pub use model_registry::ServedModelName;pub use model_registry::ServedModelRegistry;pub use model_registry::ServedModelRegistryError;
Modules§
- axum_
server - Axum-based HTTP server implementation for Ferrum
- chat_
template - middleware
- Middleware configuration types
- model_
registry - openai
- OpenAI API compatibility types
- traits
- Core server traits
- types
- Type definitions for HTTP server