Abaddon

"The Destroyer renders judgment"

Abaddon is the core inference engine for the Infernum ecosystem. It provides LLM inference with support for multiple backends and optimizations for throughput and latency.

Features

• Multi-Backend Support: CUDA, Metal, WebGPU, and CPU backends
• PagedAttention: Efficient KV-cache memory management
• FlashAttention: Fused attention kernels for speedup
• Continuous Batching: Dynamic request batching
• Speculative Decoding: Draft model acceleration
• In-Situ Quantization: Runtime INT4/INT8 conversion

Example

use abaddon::{Engine, EngineConfig};
use infernum_core::{GenerateRequest, SamplingParams};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let config = EngineConfig::builder()
        .model("meta-llama/Llama-3.2-3B-Instruct")
        .device(DeviceType::Cuda { device_id: 0 })
        .build()?;

    let engine = Engine::new(config).await?;

    let request = GenerateRequest::new("Hello, world!")
        .with_sampling(SamplingParams::balanced().with_max_tokens(100));

    let response = engine.generate(request).await?;
    println!("{}", response.choices[0].text);

    Ok(())
}