# RuvLLM v2.3 - High-Performance LLM Inference for Rust
RuvLLM is a production-ready Rust LLM inference engine optimized for Apple Silicon (M1-M4), featuring real-time fine-tuning, NEON SIMD acceleration, Apple Neural Engine integration, and the SONA self-optimizing neural architecture.
## What's New in v2.3
### Major Features
| **RuvLTRA-Medium 3B** | Purpose-built 3B model for Claude Flow | 42 layers, 256K context, speculative decode |
| **HuggingFace Hub** | Full Hub integration (download/upload) | Easy model sharing & distribution |
| **Task-Specific LoRA** | 5 pre-trained adapters for agent types | Optimized for coder/researcher/security/architect/reviewer |
| **Adapter Merging** | TIES, DARE, SLERP, Task Arithmetic | Combine adapters for multi-task models |
| **Hot-Swap Adapters** | Zero-downtime adapter switching | Runtime task specialization |
| **Claude Dataset** | 2,700+ Claude-style training examples | Optimized for Claude Flow integration |
| **HNSW Routing** | 150x faster semantic pattern matching | <25µs pattern retrieval |
| **Evaluation Harness** | Real model evaluation with SWE-Bench | 5 ablation modes, quality metrics |
| **HNSW Auto-Dimension** | Automatic embedding dimension detection | No manual config needed |
| **mistral-rs Backend** | Production-scale serving with PagedAttention | 5-10x concurrent users, X-LoRA, ISQ |
### Previous v2.0-2.2 Features
| **Apple Neural Engine** | Core ML backend with ANE routing | 38 TOPS, 3-4x power efficiency |
| **Hybrid GPU+ANE Pipeline** | Intelligent operation routing | Best of both accelerators |
| **Multi-threaded GEMM** | Rayon parallelization | 4-12x speedup on M4 Pro |
| **Flash Attention 2** | Auto block sizing, online softmax | O(N) memory, +10% throughput |
| **Quantized Inference** | INT8/INT4/Q4_K/Q8_K kernels | 4-8x memory reduction |
| **Metal GPU Shaders** | simdgroup_matrix operations | 3x speedup on Apple Silicon |
| **GGUF Support** | Memory-mapped model loading | Fast loading, reduced RAM |
| **Continuous Batching** | Dynamic batch scheduling | 2-3x throughput improvement |
| **Speculative Decoding** | Draft model acceleration | 2-3x faster generation |
| **Gemma-2 & Phi-3** | New model architectures | Extended model support |
## Features
### Multiple Backends
- **Candle Backend**: HuggingFace's Candle framework with Metal/CUDA GPU acceleration
- **Core ML Backend**: Apple Neural Engine for maximum efficiency on Apple Silicon
- **Hybrid Pipeline**: Automatic routing between GPU and ANE based on operation type
### Optimized Kernels
- **NEON SIMD**: ARM64-optimized kernels with 4x loop unrolling and FMA instructions
- **Flash Attention 2**: Memory-efficient attention with O(N) complexity and online softmax
- **Paged Attention**: Efficient KV cache management for long-context inference
- **ANE Operations**: GELU, SiLU, softmax, layer norm optimized for Neural Engine
### Real-Time Learning (SONA)
- **MicroLoRA**: Per-request fine-tuning with rank 1-2 adapters (<1ms latency)
- **EWC++**: Elastic Weight Consolidation to prevent catastrophic forgetting
- **Three-Tier Learning**: Instant (<1ms), Background (~100ms), Deep (minutes)
### Memory Efficiency
- **Two-Tier KV Cache**: FP16 tail + Q4/Q8 quantized store
- **Grouped-Query Attention (GQA)**: 4-8x KV memory reduction
- **Memory Pool**: Arena allocator for zero-allocation inference
- **GGUF Memory Mapping**: Efficient large model loading
## Quick Start
```rust
use ruvllm::prelude::*;
// Initialize backend with Metal GPU + ANE hybrid
let mut backend = CandleBackend::with_device(DeviceType::Metal)?;
// Load a GGUF model
backend.load_gguf("models/qwen2.5-7b-q4_k.gguf", ModelConfig::default())?;
// Or load from HuggingFace
backend.load_model("Qwen/Qwen2.5-7B-Instruct", ModelConfig {
quantization: Quantization::Q4K,
use_flash_attention: true,
..Default::default()
})?;
// Generate text
let response = backend.generate("Explain quantum computing in simple terms.",
GenerateParams {
max_tokens: 256,
temperature: 0.7,
top_p: 0.9,
..Default::default()
}
)?;
println!("{}", response);
// Check SONA learning stats
if let Some(stats) = backend.sona_stats() {
println!("Patterns learned: {}", stats.patterns_learned);
println!("Quality improvement: {:.1}%", stats.quality_improvement * 100.0);
}
```
## Installation
Add to your `Cargo.toml`:
```toml
[dependencies]
# Recommended for Apple Silicon Mac
ruvllm = { version = "2.0", features = ["inference-metal", "coreml", "parallel"] }
# For NVIDIA GPUs
ruvllm = { version = "2.0", features = ["inference-cuda", "parallel"] }
# Minimal (CPU only)
ruvllm = { version = "2.0" }
```
### Feature Flags
| `candle` | Enable Candle backend (HuggingFace) |
| `metal` | Apple Silicon GPU acceleration via Candle |
| `metal-compute` | Native Metal compute shaders (M4 Pro optimized) |
| `cuda` | NVIDIA GPU acceleration |
| `coreml` | Apple Neural Engine via Core ML |
| `hybrid-ane` | GPU+ANE hybrid pipeline (recommended for Mac) |
| `inference-metal` | Full Metal inference stack |
| `inference-metal-native` | Metal + native shaders (best M4 Pro perf) |
| `inference-cuda` | Full CUDA inference stack |
| `parallel` | Multi-threaded GEMM/GEMV with Rayon |
| `accelerate` | Apple Accelerate BLAS (~2x GEMV speedup) |
| `gguf-mmap` | Memory-mapped GGUF loading |
| `async-runtime` | Tokio async support |
| `wasm` | WebAssembly support |
| `mistral-rs` | mistral-rs backend (PagedAttention, X-LoRA, ISQ) |
| `mistral-rs-metal` | mistral-rs with Apple Silicon acceleration |
| `mistral-rs-cuda` | mistral-rs with NVIDIA CUDA acceleration |
## Architecture
```
+----------------------------------+
|
+----------------------------------+
| | Hybrid Pipeline Router | |
| | ┌─────────┐ ┌──────────┐ | |
| | │ Metal │ │ ANE │ | |
| | │ GPU │ │ Core ML │ | |
| | └────┬────┘ └────┬─────┘ | |
| | │ ↕ │ | |
| | Attention MLP/FFN | |
| | RoPE Activations | |
| | Softmax LayerNorm | |
| +----------------------------+ |
| | |
| +----------------------------+ |
| | SONA Learning | |
| | - Instant (<1ms) | |
| | - Background (~100ms) | |
| | - Deep (minutes) | |
| +----------------------------+ |
| | |
| +----------------------------+ |
| | NEON/SIMD Kernels | |
| | - Flash Attention 2 | |
| | - Paged KV Cache | |
| | - Quantized MatMul | |
| +----------------------------+ |
+----------------------------------+
```
## Supported Models
| **RuvLTRA-Small** | 0.5B | Q4K, Q5K, Q8, FP16 | Candle/Metal/ANE |
| **RuvLTRA-Medium** | 3B | Q4K, Q5K, Q8, FP16 | Candle/Metal |
| Qwen 2.5 | 0.5B-72B | Q4K, Q8, FP16 | Candle/Metal |
| Llama 3.x | 8B-70B | Q4K, Q8, FP16 | Candle/Metal |
| Mistral | 7B-22B | Q4K, Q8, FP16 | Candle/Metal |
| Phi-3 | 3.8B-14B | Q4K, Q8, FP16 | Candle/Metal |
| Gemma-2 | 2B-27B | Q4K, Q8, FP16 | Candle/Metal |
### RuvLTRA Models (Claude Flow Optimized)
| RuvLTRA-Small | 494M | 896 | 24 | 32K | GQA 7:1, SONA hooks |
| RuvLTRA-Medium | 3.0B | 2560 | 42 | 256K | Flash Attention 2, Speculative Decode |
## Performance (M4 Pro 14-core)
### Inference Benchmarks
| Qwen2.5-7B | Q4K | 2,800 | 95 | 4.2 GB |
| Qwen2.5-7B | Q8 | 2,100 | 72 | 7.8 GB |
| Llama3-8B | Q4K | 2,600 | 88 | 4.8 GB |
| Mistral-7B | Q4K | 2,500 | 85 | 4.1 GB |
| Phi-3-3.8B | Q4K | 3,500 | 135 | 2.3 GB |
| Gemma2-9B | Q4K | 2,200 | 75 | 5.2 GB |
### ANE vs GPU Performance (M4 Pro)
| < 512 | +30-50% | - | ANE |
| 512-1024 | +10-30% | - | ANE |
| 1024-1536 | ~Similar | ~Similar | Either |
| 1536-2048 | - | +10-20% | GPU |
| > 2048 | - | +30-50% | GPU |
### Kernel Benchmarks
| GEMM 4096x4096 | 1.2 GFLOPS | 12.7 GFLOPS |
| GEMV 4096x4096 | 0.8 GFLOPS | 6.4 GFLOPS |
| Flash Attention (seq=2048) | 850μs | 320μs |
| RMS Norm (4096) | 2.1μs | 0.8μs |
| RoPE (4096, 128) | 4.3μs | 1.6μs |
## Apple Neural Engine (ANE) Integration
RuvLLM v2.0 includes full ANE support via Core ML:
```rust
use ruvllm::backends::coreml::{CoreMLBackend, AneStrategy};
// Create ANE-optimized backend
let backend = CoreMLBackend::new(AneStrategy::PreferAneForMlp)?;
// Or use hybrid pipeline for best performance
use ruvllm::backends::HybridPipeline;
let pipeline = HybridPipeline::new(HybridConfig {
ane_strategy: AneStrategy::Adaptive,
gpu_for_attention: true, // Attention on GPU
ane_for_mlp: true, // MLP/FFN on ANE
..Default::default()
})?;
```
### ANE Routing Recommendations
| Attention | GPU | Better for variable sequence lengths |
| Flash Attention | GPU | GPU memory bandwidth advantage |
| MLP/FFN | ANE | Optimal for fixed-size matmuls |
| GELU/SiLU | ANE | Dedicated activation units |
| LayerNorm/RMSNorm | ANE | Good for small dimensions |
| Embedding | GPU | Sparse operations |
## MicroLoRA Real-Time Adaptation
RuvLLM supports per-request fine-tuning using MicroLoRA:
```rust
use ruvllm::lora::{MicroLoRA, MicroLoraConfig, AdaptFeedback};
// Create MicroLoRA adapter
let config = MicroLoraConfig::for_hidden_dim(4096);
let lora = MicroLoRA::new(config);
// Adapt on user feedback
let feedback = AdaptFeedback::from_quality(0.9);
lora.adapt(&input_embedding, feedback)?;
// Apply learned updates
lora.apply_updates(0.01); // learning rate
// Get adaptation stats
let stats = lora.stats();
println!("Samples: {}, Avg quality: {:.2}", stats.samples, stats.avg_quality);
```
## SONA Three-Tier Learning
Continuous improvement with three learning loops:
```rust
use ruvllm::optimization::{SonaLlm, SonaLlmConfig, ConsolidationStrategy};
let config = SonaLlmConfig {
instant_lr: 0.01,
background_interval_ms: 100,
deep_trigger_threshold: 100.0,
consolidation_strategy: ConsolidationStrategy::EwcMerge,
..Default::default()
};
let sona = SonaLlm::new(config);
// 1. Instant Loop (<1ms): Per-request MicroLoRA
let result = sona.instant_adapt("user query", "model response", 0.85);
println!("Instant adapt: {}μs", result.latency_us);
// 2. Background Loop (~100ms): Pattern consolidation
if let result = sona.maybe_background() {
if result.applied {
println!("Consolidated {} samples", result.samples_used);
}
}
// 3. Deep Loop (minutes): Full optimization
if sona.should_trigger_deep() {
let result = sona.deep_optimize(OptimizationTrigger::QualityThreshold(100.0));
println!("Deep optimization: {:.1}s", result.latency_us as f64 / 1_000_000.0);
}
// Check learning stats
let stats = sona.stats();
println!("Total samples: {}", stats.total_samples);
println!("Accumulated quality: {:.2}", stats.accumulated_quality);
```
## Two-Tier KV Cache
Memory-efficient caching with automatic tiering:
```rust
use ruvllm::kv_cache::{TwoTierKvCache, KvCacheConfig};
let config = KvCacheConfig {
tail_length: 256, // Recent tokens in FP16
tail_precision: Precision::FP16,
store_precision: Precision::Q4, // Older tokens in Q4
max_tokens: 8192,
num_layers: 32,
num_kv_heads: 8,
head_dim: 128,
};
let cache = TwoTierKvCache::new(config);
cache.append(&keys, &values)?;
// Automatic migration from tail to quantized store
let stats = cache.stats();
println!("Tail: {} tokens, Store: {} tokens", stats.tail_tokens, stats.store_tokens);
println!("Compression ratio: {:.2}x", stats.compression_ratio);
println!("Memory saved: {:.1} MB", stats.memory_saved_mb);
```
## Continuous Batching
High-throughput serving with dynamic batching:
```rust
use ruvllm::serving::{ContinuousBatchScheduler, SchedulerConfig, InferenceRequest};
let scheduler = ContinuousBatchScheduler::new(SchedulerConfig {
max_batch_size: 32,
max_batch_tokens: 4096,
max_waiting_time_ms: 50,
preemption_mode: PreemptionMode::Recompute,
..Default::default()
});
// Add requests
scheduler.add_request(InferenceRequest::new(tokens, params))?;
// Process batch
while let Some(batch) = scheduler.get_next_batch() {
let outputs = backend.forward_batch(&batch)?;
scheduler.process_outputs(outputs)?;
}
// Get throughput stats
let stats = scheduler.stats();
println!("Throughput: {:.1} tok/s", stats.tokens_per_second);
println!("Batch utilization: {:.1}%", stats.avg_batch_utilization * 100.0);
```
## Speculative Decoding
Accelerate generation with draft models:
```rust
use ruvllm::speculative::{SpeculativeDecoder, SpeculativeConfig};
let config = SpeculativeConfig {
draft_tokens: 4, // Tokens to draft per step
acceptance_threshold: 0.8, // Min probability for acceptance
..Default::default()
};
let decoder = SpeculativeDecoder::new(
target_model,
draft_model,
config,
)?;
// Generate with speculation
let output = decoder.generate(prompt, GenerateParams {
max_tokens: 256,
..Default::default()
})?;
println!("Acceptance rate: {:.1}%", output.stats.acceptance_rate * 100.0);
println!("Speedup: {:.2}x", output.stats.speedup);
```
## GGUF Model Loading
Efficient loading with memory mapping:
```rust
use ruvllm::gguf::{GgufLoader, GgufConfig};
let loader = GgufLoader::new(GgufConfig {
mmap_enabled: true, // Memory-map for fast loading
validate_checksum: true, // Verify file integrity
..Default::default()
});
// Load model metadata
let metadata = loader.read_metadata("model.gguf")?;
println!("Model: {}", metadata.name);
println!("Parameters: {}B", metadata.parameters / 1_000_000_000);
println!("Quantization: {:?}", metadata.quantization);
// Load into backend
let tensors = loader.load_tensors("model.gguf")?;
backend.load_tensors(tensors)?;
```
## mistral-rs Backend (Production Serving)
RuvLLM v2.3 includes integration with [mistral-rs](https://github.com/EricLBuehler/mistral.rs) for production-scale LLM serving with advanced memory management.
> **Note**: The mistral-rs crate is not yet published to crates.io. The integration is designed and ready—enable it when mistral-rs becomes available.
### Key Features
| **PagedAttention** | vLLM-style KV cache management | 5-10x concurrent users, 85-95% memory utilization |
| **X-LoRA** | Per-token adapter routing | <1ms routing overhead, multi-task inference |
| **ISQ** | In-Situ Quantization (AWQ, GPTQ, RTN) | Runtime quantization without re-export |
### Usage Example
```rust
use ruvllm::backends::mistral::{
MistralBackend, MistralBackendConfig,
PagedAttentionConfig, XLoraConfig, IsqConfig
};
// Configure mistral-rs backend for production serving
let config = MistralBackendConfig::builder()
// PagedAttention: Enable 50+ concurrent users
.paged_attention(PagedAttentionConfig {
block_size: 16,
max_blocks: 4096,
gpu_memory_fraction: 0.9,
enable_prefix_caching: true,
})
// X-LoRA: Per-token adapter routing
.xlora(XLoraConfig {
adapters: vec![
"adapters/coder".into(),
"adapters/researcher".into(),
],
top_k: 2,
temperature: 0.3,
})
// ISQ: Runtime quantization
.isq(IsqConfig {
bits: 4,
method: IsqMethod::AWQ,
calibration_samples: 128,
})
.build();
let mut backend = MistralBackend::new(config)?;
backend.load_model("mistralai/Mistral-7B-Instruct-v0.2", ModelConfig::default())?;
// Generate with PagedAttention + X-LoRA
let response = backend.generate("Write secure authentication code", GenerateParams {
max_tokens: 512,
temperature: 0.7,
..Default::default()
})?;
```
### When to Use mistral-rs vs Candle
| Single user / Edge | Candle | Simpler, smaller binary |
| 10-100 concurrent users | mistral-rs | PagedAttention memory efficiency |
| Multi-task models | mistral-rs | X-LoRA per-token routing |
| Runtime quantization | mistral-rs | ISQ without model re-export |
| WASM / Browser | Candle | mistral-rs doesn't support WASM |
### Feature Flags
```toml
# Enable mistral-rs (when available on crates.io)
ruvllm = { version = "2.3", features = ["mistral-rs"] }
# With Metal acceleration (Apple Silicon)
ruvllm = { version = "2.3", features = ["mistral-rs-metal"] }
# With CUDA acceleration (NVIDIA)
ruvllm = { version = "2.3", features = ["mistral-rs-cuda"] }
```
See [ADR-008: mistral-rs Integration](../../docs/adr/ADR-008-mistral-rs-integration.md) for detailed architecture decisions.
## Configuration
### Environment Variables
| `RUVLLM_CACHE_DIR` | Model cache directory | `~/.cache/ruvllm` |
| `RUVLLM_LOG_LEVEL` | Logging level | `info` |
| `RUVLLM_METAL_DEVICE` | Metal device index | `0` |
| `RUVLLM_ANE_ENABLED` | Enable ANE routing | `true` |
| `RUVLLM_SONA_ENABLED` | Enable SONA learning | `true` |
### Model Configuration
```rust
let config = ModelConfig {
max_context: 8192,
use_flash_attention: true,
quantization: Quantization::Q4K,
kv_cache_config: KvCacheConfig::default(),
rope_scaling: Some(RopeScaling::Linear { factor: 2.0 }),
sliding_window: Some(4096),
..Default::default()
};
```
## Benchmarks
Run benchmarks with:
```bash
# Attention benchmarks
cargo bench --bench attention_bench --features inference-metal
# ANE benchmarks (Mac only)
cargo bench --bench ane_bench --features coreml
# LoRA benchmarks
cargo bench --bench lora_bench
# End-to-end inference
cargo bench --bench e2e_bench --features inference-metal
# Metal shader benchmarks
cargo bench --bench metal_bench --features metal-compute
# Serving benchmarks
cargo bench --bench serving_bench --features inference-metal
```
## HuggingFace Hub Integration (v2.3)
Download and upload models to HuggingFace Hub:
```rust
use ruvllm::hub::{ModelDownloader, ModelUploader, RuvLtraRegistry, DownloadConfig};
// Download from Hub
let downloader = ModelDownloader::new(DownloadConfig::default());
let model_path = downloader.download(
"ruvector/ruvltra-small-q4km",
Some("./models"),
)?;
// Or use the registry for RuvLTRA models
let registry = RuvLtraRegistry::new();
let model = registry.get("ruvltra-medium", "Q4_K_M")?;
// Upload to Hub (requires HF_TOKEN)
let uploader = ModelUploader::new("hf_your_token");
let url = uploader.upload(
"./my-model.gguf",
"username/my-ruvltra-model",
Some(metadata),
)?;
println!("Uploaded to: {}", url);
```
## Task-Specific LoRA Adapters (v2.3)
Pre-trained adapters optimized for Claude Flow agent types:
```rust
use ruvllm::lora::{RuvLtraAdapters, AdapterTrainer, AdapterMerger, HotSwapManager};
// Create adapter for specific task
let adapters = RuvLtraAdapters::new();
let coder = adapters.create_lora("coder", 768)?; // Rank 16, code generation
let security = adapters.create_lora("security", 768)?; // Rank 16, vulnerability detection
// Available adapters:
// - coder: Rank 16, Alpha 32.0, targets attention (Q,K,V,O)
// - researcher: Rank 8, Alpha 16.0, targets Q,K,V
// - security: Rank 16, Alpha 32.0, targets attention + MLP
// - architect: Rank 12, Alpha 24.0, targets Q,V + Gate,Up
// - reviewer: Rank 8, Alpha 16.0, targets Q,V
// Merge adapters for multi-task models
let merger = AdapterMerger::new(MergeConfig::weighted(weights));
let multi_task = merger.merge(&[coder, security], &output_config, 768)?;
// Hot-swap adapters at runtime
let mut manager = HotSwapManager::new();
manager.set_active(coder);
manager.prepare_standby(security);
manager.swap()?; // Zero-downtime switch
```
### Adapter Merging Strategies
| **Average** | Equal-weight averaging | Simple multi-task |
| **WeightedSum** | User-defined weights | Task importance weighting |
| **SLERP** | Spherical interpolation | Smooth transitions |
| **TIES** | Trim, Elect, Merge | Robust multi-adapter |
| **DARE** | Drop And REscale | Sparse merging |
| **TaskArithmetic** | Add/subtract vectors | Task composition |
## Evaluation Harness (v2.3)
RuvLLM includes a comprehensive evaluation harness for benchmarking model quality:
```rust
use ruvllm::evaluation::{RealEvaluationHarness, EvalConfig, AblationMode};
// Create harness with GGUF model
let harness = RealEvaluationHarness::with_gguf(
"./models/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf",
EvalConfig::default(),
)?;
// Run single evaluation
let result = harness.evaluate(
"Fix the null pointer exception in this code",
"def process(data):\n return data.split()",
AblationMode::Full,
)?;
println!("Success: {}, Quality: {:.2}", result.success, result.quality_score);
// Run full ablation study (5 modes)
let report = harness.run_ablation_study(&tasks)?;
for (mode, metrics) in &report.mode_metrics {
println!("{:?}: {:.1}% success, {:.2} quality",
mode, metrics.success_rate * 100.0, metrics.avg_quality);
}
```
### Ablation Modes
| **Baseline** | No enhancements | Control baseline |
| **RetrievalOnly** | HNSW pattern retrieval | Measure retrieval impact |
| **AdaptersOnly** | LoRA adapters | Measure adaptation impact |
| **RetrievalPlusAdapters** | HNSW + LoRA | Combined without SONA |
| **Full** | All systems (SONA + HNSW + LoRA) | Production mode |
### SWE-Bench Task Loader
```rust
use ruvllm::evaluation::swe_bench::SweBenchLoader;
// Load SWE-Bench tasks
let loader = SweBenchLoader::new();
let tasks = loader.load_subset("lite", 50)?; // 50 tasks from lite subset
for task in &tasks {
println!("Instance: {}", task.instance_id);
println!("Problem: {}", task.problem_statement);
}
```
### CLI Evaluation
```bash
# Run evaluation with default settings
cargo run --example run_eval --features async-runtime -- \
--model ./models/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf
# Run SWE-Bench subset
cargo run --example run_eval --features async-runtime -- \
--model ./models/model.gguf \
--swe-bench-path ./data/swe-bench \
--subset lite \
--max-tasks 100
# Output report
cargo run --example run_eval --features async-runtime -- \
--model ./models/model.gguf \
--output ./reports/eval-report.json
```
### HNSW Auto-Dimension Detection
The evaluation harness automatically detects model embedding dimensions:
```rust
// HNSW router automatically uses model's hidden_size
// TinyLlama 1.1B → 2048 dimensions
// Qwen2 0.5B → 896 dimensions
// RuvLTRA-Small → 896 dimensions
// RuvLTRA-Medium → 2560 dimensions
let harness = RealEvaluationHarness::with_config(
EvalConfig::default(),
RealInferenceConfig {
enable_hnsw: true,
hnsw_config: None, // Auto-detect from model
..Default::default()
},
)?;
```
## Examples
See the `/examples` directory for:
- `download_test_model.rs` - Download and validate models
- `benchmark_model.rs` - Full inference benchmarking
- `run_eval.rs` - Run evaluation harness with SWE-Bench
- Basic inference
- Streaming generation
- MicroLoRA adaptation
- Multi-turn chat
- Speculative decoding
- Continuous batching
- ANE hybrid inference
## Error Handling
```rust
use ruvllm::error::{Result, RuvLLMError};
match backend.generate(prompt, params) {
Ok(response) => println!("{}", response),
Err(RuvLLMError::Model(e)) => eprintln!("Model error: {}", e),
Err(RuvLLMError::OutOfMemory(e)) => eprintln!("OOM: {}", e),
Err(RuvLLMError::Generation(e)) => eprintln!("Generation failed: {}", e),
Err(RuvLLMError::Ane(e)) => eprintln!("ANE error: {}", e),
Err(RuvLLMError::Gguf(e)) => eprintln!("GGUF loading error: {}", e),
Err(e) => eprintln!("Error: {}", e),
}
```
## npm Package
RuvLLM is also available as an npm package with native bindings:
```bash
npm install @ruvector/ruvllm
```
```typescript
import { RuvLLM } from '@ruvector/ruvllm';
const llm = new RuvLLM();
const response = llm.query('Explain quantum computing');
console.log(response.text);
```
See [@ruvector/ruvllm on npm](https://www.npmjs.com/package/@ruvector/ruvllm) for full documentation.
## License
Apache-2.0 / MIT dual license.
## Contributing
Contributions welcome! Please see [CONTRIBUTING.md](../../CONTRIBUTING.md) for guidelines.
## Links
- [GitHub Repository](https://github.com/ruvnet/ruvector)
- [API Documentation](https://docs.rs/ruvllm)
- [npm Package](https://www.npmjs.com/package/@ruvector/ruvllm)
- [Issue Tracker](https://github.com/ruvnet/ruvector/issues)