ForgeLLM

Compile your LLMs, don't interpret them.

ForgeLLM is a Rust-native ahead-of-time (AOT) ML compiler for language models (1M-7B parameters). It compiles GGUF models into optimized, self-contained binaries with native Metal GPU acceleration — no runtime interpreter, no Python dependencies, no dynamic dispatch.

Faster than llama.cpp on Apple Silicon.

Documentation | Crates.io | forgellm.dev | Blog: How we beat llama.cpp

Performance

Benchmarks on Apple M5 Pro, 8-bit quantization, 64-token generation.

Generation Speed (tok/s)

Prefill Speed (tok/s, long prompt)

Deploy Size

Binary size is constant across all models. Compare: llama.cpp ~15 MB, MLX ~500 MB Python runtime.

We beat MLX and llama.cpp on generation across all model sizes, and on prefill for small-to-medium models. For very large models (1B+), MLX's Apple Accelerate BLAS leads on prefill — closing that gap requires hardware matrix multiply instructions (simdgroup_multiply_accumulate).

See benchmarks/HISTORY.md and blog/beating-llama-cpp.md for details.

Quick Start

Metal GPU (Apple Silicon)

# Build from source
git clone https://github.com/sauravpanda/forge-llm.git
cd forge-llm && cargo build --release

# Compile model to Metal binary
forge compile --model model.gguf --output ./my-model --target metal
forge export-weights --model model.gguf --output ./my-model/weights.bin
cp tokenizer.json ./my-model/

# Build and run
cd my-model && cargo build --release
./target/release/my-model weights.bin tokenizer.json "The meaning of life is"

API Server

# Start OpenAI-compatible server
./target/release/my-model weights.bin tokenizer.json --serve --port 8080

# Query it
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Hello!"}], "stream": true}'

CPU (cross-platform)

# Compile for CPU with NEON SIMD + Rayon parallelism
forge compile --model model.gguf --output ./my-model --target cpu --run

Why ForgeLLM is faster

Every existing LLM inference engine — llama.cpp, vLLM, MLX — loads model weights at runtime and executes a generic inference loop. This is like shipping a Python interpreter when you could ship a compiled binary.

ForgeLLM compiles models into hardware-specific code:

Compilation Targets

Supported Models

Supports GGUF quantization formats: F32, F16, BF16, Q8_0, Q4_0, Q4_1, Q2_K through Q8_K. Also supports SafeTensors and LoRA adapter merging at compile time.

Metal GPU Features

The Metal backend generates optimized Apple Silicon compute shaders:

• Simdgroup cooperative matmul — 32-lane SIMD reductions with shared memory vector caching
• Native Q8_0/Q4_0 kernels — Dequantize on-the-fly during matmul, halving memory bandwidth
• Fused projections — QKV and gate+up concatenated into single matmul dispatches
• Single compute encoder — Entire forward pass in one encoder, zero transitions
• Double-buffered prefill — GPU overlaps with CPU encoding
• fast:: math — Hardware-accelerated rsqrt/exp in normalization and attention
• OpenAI-compatible API — --serve mode with SSE streaming

CLI Commands

# AOT compile to Metal GPU binary
forge compile --model model.gguf --output ./out --target metal

# AOT compile to CPU binary
forge compile --model model.gguf --output ./out --target cpu --run

# Export weights for compiled binary
forge export-weights --model model.gguf --output ./out/weights.bin

# Run interpreter (no compilation)
forge run --model model.gguf --tokenizer tokenizer.json --prompt "Hello"

# Interactive chat
forge chat --model model.gguf --tokenizer tokenizer.json

# Start API server (interpreter mode)
forge serve --model model.gguf --tokenizer tokenizer.json --port 8080

# Benchmark
forge bench --model model.gguf --tokenizer tokenizer.json --num-tokens 128

# Inspect model
forge info model.gguf

# ONNX export
forge export-onnx --model model.gguf --output model.onnx

# Speculative decoding
forge speculative --draft small.gguf --target-model large.gguf --output ./spec

Architecture

GGUF/SafeTensors → Frontend → IR Graph → Optimizer → Codegen → Binary
                     parse      build      fuse       emit     compile

8 crates: forgellm-frontend, forgellm-optimizer, forgellm-codegen-cpu, forgellm-codegen-wasm, forgellm-codegen-gpu, forgellm-codegen-metal, forgellm-runtime, forgellm-cli

Contributing

cargo test --workspace --exclude forgellm-python  # 258+ tests
cargo clippy --workspace -- -D warnings
cargo fmt --all -- --check

License

MIT