realizar 0.8.6

Table of Contents

• What is realizar?
• Installation
• Usage
• Features
• Benchmarks
• Quality
• Sovereign AI Stack
• Documentation
• Contributing
• License

What is realizar?

realizar is a pure Rust LLM inference engine. It loads models in APR v2, GGUF, and SafeTensors formats, runs transformer inference with quantized kernels (Q4_K through Q8_0), and serves predictions over an OpenAI-compatible REST API.

Key design decisions:

• Row-major mandate -- All tensors are row-major internally. GGUF column-major data is transposed at import by aprender. This matches PyTorch/SafeTensors layout and simplifies kernel implementations.
• Pure Rust CUDA -- GPU acceleration via trueno-gpu generates PTX directly from Rust. No nvcc, no LLVM, no C++ dependencies.
• Cost-based dispatch -- Backend selection (GPU/SIMD/scalar) uses a 5x PCIe cost model to avoid GPU overhead on small workloads.

Installation

CLI

cargo install realizar

Library

Add to your Cargo.toml:

[dependencies]
realizar = "0.8"

Usage

Serving

# Start demo server
realizar serve --demo --port 8080

# Health check
curl http://localhost:8080/health

# Prometheus metrics
curl http://localhost:8080/metrics

OpenAI-Compatible API

# Chat completions
curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "default",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 50
  }'

# Streaming
curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "default",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": true
  }'

Library API

use realizar::chat_template::{auto_detect_template, ChatMessage};

let template = auto_detect_template("Qwen2-0.5B-Instruct");
let messages = vec![
    ChatMessage::system("You are a helpful assistant."),
    ChatMessage::user("Hello!"),
];
let formatted = template.format_conversation(&messages)?;

Tracing

Use the X-Trace-Level header for inference debugging:

# Brick-level: token-by-token timing
curl -H "X-Trace-Level: brick" -X POST http://localhost:8080/v1/chat/completions ...

# Layer-level: per-layer timing breakdown
curl -H "X-Trace-Level: layer" -X POST http://localhost:8080/v1/chat/completions ...

Features

Model Formats

GPU Kernels

Quantization

Q4_0, Q8_0, Q4_K, Q5_K, Q6_K -- SIMD-accelerated on AVX2, AVX-512, and NEON. GPU dequantization fused with matrix operations to avoid memory round-trips.

KV Cache

Autoregressive decoding with persistent key-value cache. Supports grouped-query attention (GQA) for models like Qwen2.5 and Llama 3.

Chat Templates

Automatic template detection from model metadata:

Feature Flags

Benchmarks

LLM Inference (GPU)

realizar achieves 8--21% faster inference than llama.cpp/Ollama via pure Rust CUDA PTX generation.

Classical ML (APR Format)

Methodology follows Hoefler & Belli SC'15 (CV-based stopping, warmup iterations discarded).

Quality

• 15,000+ tests across unit, integration, and property-based suites
• 95%+ line coverage via cargo-llvm-cov
• Zero clippy warnings with -D warnings
• Mutation testing via cargo-mutants
• Provable contracts -- 1,725 bindings with AllImplemented policy

Sovereign AI Stack

realizar is the inference layer of the PAIML Sovereign AI Stack:

Documentation

• API docs: docs.rs/realizar
• Repository: github.com/paiml/realizar
• Cookbook: github.com/paiml/apr-cookbook

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines, or open an issue to discuss your idea first.

License

MIT -- Pragmatic AI Labs