Inference Lab

High-performance LLM inference simulator for analyzing serving systems. Simulates GPU clusters serving LLM inference workloads with realistic performance modeling.

Features

Accurate Performance Modeling: Models compute (FLOPS) and memory bandwidth constraints
Multiple Scheduling Policies: FCFS, Priority, SJF, and more
Chunked Prefill: Simulates realistic request interleaving
KV Cache Management: Models GPU memory and KV cache utilization
Workload Generation: Supports Poisson, Gamma, and closed-loop patterns
WebAssembly Support: Run simulations in the browser via WASM
CLI Tool: Standalone binary for command-line usage

Installation

As a Rust Library

cargo add inference-lab

As an npm Package (WASM)

npm install @doublewordai/inference-lab

CLI Tool

cargo install inference-lab

Usage

CLI

# Run with default configuration
inference-lab --config configs/config.toml

# Example output shows TTFT, E2E latency, throughput, and utilization metrics

Rust Library

use inference_lab::simulation::Simulator;
use inference_lab::config::SimulationConfig;

let config = SimulationConfig::from_file("config.toml")?;
let mut simulator = Simulator::new(config);
let results = simulator.run();

println!("Mean TTFT: {:.2}ms", results.ttft_mean * 1000.0);
println!("P99 E2E: {:.2}ms", results.e2e_p99 * 1000.0);
println!("Throughput: {:.1} tok/s", results.throughput);

WebAssembly

import init, { run_simulation } from '@doubleword/inference-lab';

await init();

const config = `
[hardware]
name = "H100"
compute_flops = 2e15
memory_bandwidth = 3.35e12
# ... rest of config
`;

const results = run_simulation(config);
console.log('TTFT P50:', results.ttft_p50);
console.log('Throughput:', results.throughput);

Configuration

Configuration files use TOML format and specify:

Hardware: GPU specs (FLOPS, bandwidth, VRAM)
Model: LLM architecture (parameters, layers, heads)
Scheduler: Policies, max tokens, chunked prefill settings
Workload: Request arrival patterns and distributions

Example configurations are in the configs/ directory:

config.toml - Default H100 + Llama-3-70B setup
test_blog.toml - Closed-loop benchmark (64 users)
qwen3_30b_a3b.toml - Qwen model configuration

Building

Native Binary

cargo build --release
./target/release/inference-lab --config configs/config.toml

WASM Package

npm run build
# Outputs to pkg/ directory

Publishing

# Publish to npm (requires authentication)
npm run build
npm publish --access public

# Publish Rust crate
cargo publish

Project Structure

inference-lab/
├── src/
│   ├── simulation/     # Core simulator logic
│   ├── scheduler/      # Scheduling policies (FCFS, Priority, SJF)
│   ├── compute/        # Performance calculations
│   ├── kv_cache/       # KV cache management
│   ├── request/        # Request generation and tracking
│   ├── metrics/        # Performance metrics collection
│   ├── config/         # Configuration structures
│   ├── lib.rs          # Library root
│   ├── main.rs         # CLI entry point
│   └── wasm.rs         # WebAssembly bindings
├── configs/            # Example configurations
├── Cargo.toml          # Rust package manifest
└── package.json        # npm package manifest

Metrics

The simulator tracks:

TTFT (Time to First Token): Prefill latency
E2E (End-to-End): Total request latency
TPOT (Time Per Output Token): Decode latency per token
Throughput: Tokens generated per second
Utilization: Compute and memory bandwidth usage
KV Cache: Memory utilization over time

Results include percentiles (p50, p90, p95, p99) and means.

License

MIT

Repository

https://github.com/doublewordai/inference-lab

inference-lab 0.1.1