Crate ferrum_engine 

Ferrum Engine

LLM inference engine orchestration layer with strong streaming support.

Overview

This crate provides the main inference engine implementation that orchestrates all the components from other ferrum crates:

• Request admission and scheduling (ferrum-scheduler)
• KV-cache allocation and management (ferrum-kv)
• Tokenization and incremental decoding (ferrum-tokenizer)
• Logits processing and sampling (ferrum-sampler)
• Model execution and weight loading (ferrum-models)
• Runtime and compute backends (ferrum-runtime)

Design Principles

• Strong Streaming: TTFT optimization and consistent inter-token latency
• Orchestration Layer: Compose components rather than implement functionality
• Batch Processing: Dynamic continuous batching for throughput
• Pipeline Optimization: Prefill→decode loops with minimal overhead
• Registry Pattern: Dynamic component registration and lookup

Usage

Using the Engine Builder (Recommended)

use ferrum_engine::{EngineBuilder, EngineConfig};

let config = EngineConfig::default();
let engine = EngineBuilder::new(config)
    .with_scheduler("fifo")
    .with_sampler("greedy")
    .build()
    .await?;

Registering Custom Components

use ferrum_engine::{ComponentRegistry, global_registry};

let registry = global_registry();
registry.register_backend_factory("my_backend", Arc::new(MyBackendFactory));

Re-exports

pub use continuous_engine::ContinuousBatchEngine;

pub use continuous_engine::SequenceState;

pub use pipeline::ChunkedPrefillConfig;

pub use pipeline::ChunkedPrefillExecutor;

pub use pipeline::ExecutionPhase;

pub use pipeline::PipelineConfig;

pub use pipeline::PipelineExecutor;

pub use recurrent_state::InMemoryRecurrentStateConfig;

pub use recurrent_state::InMemoryRecurrentStateHandle;

pub use recurrent_state::InMemoryRecurrentStateManager;

pub use builder::create_engine;

pub use builder::create_prepared_product_engine;

pub use builder::create_product_engine;

pub use builder::EngineBuilder;

pub use registry::global_registry;

pub use registry::set_global_registry;

pub use registry::ComponentConfig;

pub use registry::ComponentFactory;

pub use registry::ComponentMetadata;

pub use registry::ComponentRegistry;

pub use registry::ContinuousBatchSchedulerFactory;

pub use registry::DefaultKvCacheFactory;

pub use registry::FifoSchedulerFactory;

pub use registry::GreedySamplerFactory;

pub use registry::HuggingFaceTokenizerFactory;

pub use registry::LlmExecutorFactory;

pub use registry::MultinomialSamplerFactory;

pub use registry::PagedKvCacheFactory;

pub use registry::PrioritySchedulerFactory;

pub use registry::StubExecutorFactory;

pub use registry::StubTokenizer;

pub use registry::StubTokenizerFactory;

pub use registry::CandleExecutorFactory;Deprecated

pub use parallel::global_device_manager;

pub use parallel::DeviceCapability;

pub use parallel::DeviceInfo;

pub use parallel::DeviceManager;

pub use parallel::LayerDistribution;

pub use parallel::ParallelConfig;

pub use parallel::ParallelExecutor;

pub use parallel::ParallelExecutorFactory;

pub use parallel::ParallelismType;

pub use parallel::TensorParallelConfig;

pub use parallel::TensorParallelGroup;

Modules

builder

Engine builder with registry-based component creation

continuous_engine

Continuous Batching Engine

embedding_engine

Lightweight engine for embedding models (CLIP, BERT, etc.).

modality_stubs

Shared boilerplate for modality-only engines (embedding / transcription / TTS).

parallel

Multi-GPU Parallelism Module

pipeline

Inference Pipeline Implementations

recurrent_state

In-memory recurrent-state manager used for lifecycle and integration tests.

registry

Component registry for dynamic component creation

speculative

Speculative decoding — draft + verify.

tensor_factory

Candle-tied TensorFactory impl used by the registry’s stub-executor factory.

transcription_engine

Lightweight engine for Whisper ASR transcription.

tts_engine

TTS service — concurrent slot-based serving of TtsModelExecutor.

Structs

BatchId

Batch identifier

BatchPlan

Batch execution plan

EngineConfig

Engine configuration

EngineStatus

Engine status information

GreedySampler

Common samplers Greedy sampler (always picks highest probability token)

InferenceRequest

Inference request

InferenceResponse

Inference response

RequestId

Request identifier

StreamChunk

Streaming response chunk

Enums

FerrumError

Main error type for Ferrum operations

Traits

EmbedEngine

Embedding engine (CLIP, BERT, etc.).

IncrementalTokenizer

Incremental tokenizer state for streaming

InferenceEngineInterface

Lifecycle / status methods shared by every engine kind.

KvCacheManager

KV cache manager for allocation and lifecycle management

LlmInferenceEngine

LLM text-generation engine.

ModelExecutor

Core model executor trait focusing on tensor operations

Sampler

Token sampler trait for selecting next token from processed logits

Scheduler

Main scheduler trait for request management and batching

Tokenizer

Core tokenizer trait for encoding/decoding operations

TranscribeEngine

Speech-to-text (Whisper) engine.

TtsEngine

Text-to-speech (Qwen3-TTS, etc.) engine.

Functions

create_default_engine

Create default inference engine with MVP configuration

Type Aliases

Result

Result type used throughout Ferrum