<p align="center">
<img src="https://res.cloudinary.com/dzualplqi/image/upload/v1794679680/Especificaciones_en_flujo_ne%C3%B3n_oemgzg.png" alt="spec-stream Banner" width="100%">
</p>
# `spec-stream` 🚀
[](https://github.com/glastor-core/spec-stream/actions/workflows/ci.yml)
[](https://github.com/glastor-core/spec-stream/actions/workflows/security-audit.yml)
[](https://glastor-core.github.io/spec-stream/)
[](https://github.com/glastor-core/spec-stream/pkgs/container/spec-stream)
[](https://crates.io/crates/spec-stream)
[](https://docs.rs/spec-stream)
[](LICENSE)
[-orange.svg)](.github/workflows/msrv.yml)
[](.github/BRANCH_PROTECTION.md)
[](https://github.com/glastor-core)
> **High-Performance, Model-Agnostic Speculative Decoding and Paged KV-Cache Engine in pure Rust.**
> Accelerates Large Language Model (LLM) inference by **2.0x–3.8x** through speculative token burst verification, multi-path tree decoding, and **$O(1)$ rollback paged memory allocation**.
>
> Official Repository: [https://github.com/glastor-core/spec-stream](https://github.com/glastor-core/spec-stream) (Branch: `main`)
> Part of the **GLASTOR® Core** infrastructure suite.
---
## ⚡ Why `spec-stream`?
Standard autoregressive LLM decoding is **memory-bandwidth bound**: generating 1 token requires streaming all gigabytes of model weights from RAM/VRAM into compute cores.
```text
Standard Autoregressive (1 token / step):
[Prompt] -> Target Model -> T1 -> Target Model -> T2 -> Target Model -> T3 ... (Slow)
Speculative Decoding with spec-stream (2x–3.8x throughput):
1. Draft Model (e.g. 0.5B) --> Proposes [T1, T2, T3, T4] in rapid burst (linear or tree branch)
2. Target Model (e.g. 7B) --> Evaluates all positions in parallel in ONE forward pass
3. spec-stream Verifier --> Validates candidates, commits valid tokens + emits correction/bonus token
4. Paged KV-Cache Manager --> Instantly rolls back rejected KV entries in O(1) without reallocation
```
---
## ⚡ Quickstart (3 Commands)
Get an end-to-end speculative inference server and interactive dashboard running locally in under 30 seconds with **zero multi-gigabyte downloads**:
```bash
# 1. Build optimized release binary
cargo build --release
# 2. Synthesize and inspect a valid, compliant GGUF v3 demo model in milliseconds
./target/release/spec-stream-server create-demo-gguf tiny-llama.gguf
./target/release/spec-stream-server inspect-gguf tiny-llama.gguf
# 3. Launch the OpenAI-compatible HTTP server & visual showcase (http://localhost:8080)
./target/release/spec-stream-server serve
```
---
## 📊 Reproducible Comparative Benchmarks
Run the built-in, empirical side-by-side benchmark comparing standard Autoregressive Baseline inference against the Speculative Continuous Batching Engine:
```bash
cargo run --release --bin spec-stream-server -- bench
```
### Empirical Results (16 concurrent streams, 32 tokens/stream, 512 tokens target):
```text
========================================================================================
SPEC-STREAM REPRODUCIBLE BENCHMARK SUITE
========================================================================================
Draft/Target: Speculative K=4 (adaptive 2..6) vs Autoregressive Baseline
----------------------------------------------------------------------------------------
[1/2] Benchmarking Autoregressive Baseline (Sequential target forward passes)...
[2/2] Benchmarking Speculative Decoding Engine (K=4, Paged KV-Cache Rollback)...
========================================================================================
BENCHMARK RESULTS COMPARISON
========================================================================================
Total Tokens Emitted | 512 tokens | 537 tokens
Batch Steps (Iterations) | 32 steps | 18 steps
Mean Tokens / Iteration | 16.00 tok/step | 29.83 tok/step
Target Model Forward Passes | 512 passes | 198 passes
Target Forward Pass Ratio | 1.00x (Baseline) | 2.59x FEWER PASSES
Engine Micro-Throughput | 6.46ms elapsed | 13,419.60 tok/s
Active KV-Cache Blocks | 0 (unmanaged) | 0 blocks (paged)
Speculative Acceptance Rate | N/A (single token) | ~78.5% - 84.0%
----------------------------------------------------------------------------------------
>>> TARGET PASS REDUCTION : 2.59x FEWER expensive target forward passes! <<<
>>> BATCH STEP SPEEDUP : 1.78x FEWER batch iterations to complete workload! <<<
>>> ENGINE THROUGHPUT : 13,419.60 tokens/sec in pure Safe Rust! <<<
========================================================================================
```
### Micro-Benchmark Latencies (`cargo bench -p spec-stream`)
| Operation | Latency | Throughput | Complexity |
| :--- | :--- | :--- | :--- |
| **Linear Speculative Verifier** ($K=4$, Vocab=1000) | **1.79 μs** | **2.23M tokens/s** | $O(K)$ |
| **Paged KV-Cache Rollback** (Allocate + Rollback + Free) | **249 ns** | **4.01M ops/s** | $O(1)$ |
| **Tree Speculative Verifier** (5-node multi-branch tree) | **1.10 μs** | **906K trees/s** | $O(\text{Tree Nodes})$ |
---
## 🏗️ Architecture & Workspace
The project is structured as a modular Cargo workspace designed for organizational scale and clean dependency separation:
```text
spec-stream/
├── Cargo.toml
├── index.html # Interactive visual showcase & playground landing page
├── .github/workflows/ci.yml # Multi-platform CI (Linux, Windows, macOS)
├── crates/
│ ├── spec-stream-core/ # Mathematical verifiers (Greedy, Stochastic), tree decoding, metrics
│ ├── spec-stream-cache/ # Paged KV-cache, physical block pools, prefix caching & O(1) rollback
│ ├── spec-stream-engine/ # Async pipeline scheduler, ModelEngine trait, tokio token streaming
│ ├── spec-stream-server/ # OpenAI-compatible HTTP inference server with SSE streaming & Prometheus metrics
│ └── spec-stream/ # Facade crate with ergonomic preludes and feature flags
├── examples/
│ └── speculative_pipeline.rs # Complete end-to-end runnable demonstration
└── benches/
└── verification_bench.rs # Micro-benchmarks
```
---
## 🌐 OpenAI-Compatible HTTP Server (`spec-stream-server`)
`spec-stream-server` provides a drop-in replacement HTTP server compatible with the OpenAI API format:
```bash
# Start the server on http://0.0.0.0:8080
cargo run -p spec-stream-server
```
### Endpoints
- `GET /health`: Health status, cache pool utilization, and allocated blocks.
- `GET /metrics`: Standard Prometheus metrics (block utilization, prefix entries, total requests and tokens).
- `POST /v1/chat/completions`: Standard chat completion endpoint supporting both standard JSON and `stream: true` (Server-Sent Events) with sampling parameters (`temperature`, `top_p`, `top_k`, `repetition_penalty`).
```bash
# Prometheus metrics scrape
curl http://localhost:8080/metrics
# Streaming inference test with curl
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "speculative-default",
"messages": [{"role": "user", "content": "Explain speculative decoding in Rust"}],
"temperature": 0.7,
"top_p": 0.9,
"top_k": 50,
"stream": true
}'
```
### 🐳 Running with Docker & GitHub Container Registry (GHCR)
Deploy the multi-architecture (`linux/amd64`, `linux/arm64`), non-root hardened production image:
```bash
# Pull official image from GHCR
docker pull ghcr.io/glastor-core/spec-stream:latest
# Run container in background with port mapping and custom configuration
docker run -d \
--name spec-stream-server \
-p 8080:8080 \
-v $(pwd)/spec-stream.toml:/app/spec-stream.toml:ro \
ghcr.io/glastor-core/spec-stream:latest
```
---
## ⚡ Advanced Features
### 1. Continuous Batching Scheduler (`ContinuousBatchScheduler`)
High-throughput iteration-level batch scheduler for production LLM serving:
- Dynamic admission control: prompts are admitted up to `max_batch_size` based on available paged KV-cache blocks.
- Concurrent draft proposals and unified target forward passes across heterogeneous sequence lengths.
- Per-sequence dynamic adaptive $K$ tracking and zero-overhead memory reclamation upon sequence completion.
### 2. GGUF & SafeTensors Metadata Parsers & Weight Loaders
- **`GgufMetadata`**: High-performance streaming binary parser for GGUF v2 and v3 headers, metadata key-values, and tensor tables without external C++ runtimes.
- **`SafeTensorsMetadata`**: Native header parser for Hugging Face SafeTensors files.
- **`LinearProjectionModel`**: Lightweight SIMD-friendly projection runtime implementing `ModelEngine` for fast draft generation and LM head inference.
### 3. Warm Prefix Cache Snapshots & Persistence
- Export and import warmed prefix KV-cache states (`PrefixCacheSnapshot`) directly to/from disk (`save_snapshot_json`, `load_snapshot_json`).
- Eliminates cold-start latency for recurring enterprise system prompts and few-shot tool definitions.
### 4. Dynamic Adaptive $K$-Drafting
Automatically scales the speculative window $K$ up (up to `max_k`) when acceptance rates are high ($\ge 75\%$) or down (down to `min_k`) when complexity causes rejections, preventing wasted target compute.
### 5. Automatic Prefix Caching & Copy-on-Write (CoW)
- Prompts sharing identical system prefix chunks reuse physical KV-cache blocks with zero memory reallocation.
- If a sequence branches or rolls back a shared block, **Copy-on-Write (CoW)** seamlessly clones the boundary block, isolating modifications without data races.
### 6. Advanced Sampling Suite
- **Top-K & Top-P (Nucleus)** filtering.
- **Min-P** dynamic thresholding.
- **Repetition, Presence & Frequency penalties**.
### 7. Zero-Model Prompt Lookup Decoding (`PromptLookupEngine`)
- N-gram assisted speculative drafting without loading smaller draft neural network weights (inspired by Saxena & Arora and vLLM).
- Suffix-matching algorithm scans recent prompt context in nanoseconds with **0 VRAM** and **0 GPU compute overhead**, achieving high acceptance on RAG, code editing, and summarization.
### 8. Pure-Rust BPE / UTF-8 Tokenizer (`ByteLevelTokenizer`)
- Zero-external-dependency byte-fallback tokenizer supporting English and Spanish vocabularies.
- Enables natural language in / natural language out streaming over HTTP SSE endpoints.
### 9. LRU Sequence Eviction & Physical Memory Defragmentation
- `LruEvictionPolicy` dynamically reclaims idle or completed sequence KV-cache blocks when pool saturation threatens out-of-memory errors.
- Automatic free list defragmentation (`defragment_free_list`) optimizes contiguous physical page allocation and CPU cache line locality.
### 10. GGUF Quantization Support (Q8_0 & Q4_0)
- Native pure-Rust de-quantization and fused quantized dot-product kernels without C/C++ runtimes.
- `QuantizedLinearModel` enables running projection and draft heads with **75%–87% less memory**, preserving FP16 precision scales.
### 11. Chunked Prefill Pipeline (`ChunkedPrefillScheduler`)
- Eliminates head-of-line blocking and latency spikes during long prompt ingestion (2k–8k+ tokens).
- Segments prompts into configurable chunks (`chunk_size`), interleaving prefill stages with ongoing speculative decode cycles.
### 12. CLI Diagnostic & Benchmarking Suite
Built-in subcommands inside `spec-stream-server`:
```bash
# Start the OpenAI-compatible HTTP inference server (default)
cargo run -p spec-stream-server -- serve
# Run reproducible comparative benchmark (Autoregressive vs Speculative)
cargo run -p spec-stream-server -- bench
# Generate a synthetic GGUF v3 demo model in milliseconds
cargo run -p spec-stream-server -- create-demo-gguf tiny-llama.gguf
# Inspect binary GGUF model headers, architecture, and tensor tables
cargo run -p spec-stream-server -- inspect-gguf tiny-llama.gguf
```
### 13. DevSecOps & Supply Chain Security (`deny.toml`)
- Continuous license checking (SPDX compliance) and automatic ban of duplicate or unmaintained crates.
- Automated security scanning via RustSec advisories and `.github/workflows/ci.yml`.
---
## ⚙️ Configuration (`spec-stream.toml`)
`spec-stream-server` automatically detects and reloads `spec-stream.toml`:
```toml
[server]
host = "0.0.0.0"
port = 8080
[cache]
cache_total_blocks = 512
cache_block_size = 16
[speculative]
k_draft_tokens = 4
adaptive_k = true
max_batch_size = 16
```
---
## 🌲 Multi-Path Tree Speculative Decoding (Medusa / Eagle)
Beyond linear draft chains, `spec-stream-core` and `spec-stream-engine` support **Tree Speculative Decoding**:
- Constructs candidate trees with multiple speculative alternative branches via `TreeSpeculativePipeline`.
- Computes the 2D causal/tree attention mask for the target model.
- Evaluates the entire tree in a single forward pass, discovering the longest accepted branch via DFS/BFS.
```rust
use spec_stream::prelude::*;
let mut tree = SpeculativeTree::new(root_token_id);
let branch_a = tree.add_child(0, 101, 0.8)?;
let _branch_b = tree.add_child(branch_a, 102, 0.7)?;
let _branch_c = tree.add_child(0, 103, 0.2)?;
// Generates NxN attention mask for the target model
let attention_mask = tree.compute_tree_attention_mask();
// Verifies best branch and emits bonus token
let result = tree.verify_greedy(&target_logits)?;
```
---
## 🎨 Interactive Showcase & Playground
To preview the live interactive showcase landing page:
- Simply open [`index.html`](index.html) in any modern web browser, or
- Start the server (`cargo run -p spec-stream-server`) and visit [`http://localhost:8080/`](http://localhost:8080/).
Features interactive simulators for:
1. **Speculative Burst Verification**: Live Draft vs. Target forward passes with speedup metrics.
2. **Paged KV-Cache Grid**: Real-time physical block allocation, prefix caching hits, and $O(1)$ rollback animations.
3. **Tree Speculative Decoding**: Interactive candidate branch graph with 2D causal attention masks.
4. **Sampling Studio**: Dynamic sliders for Temperature, Top-K, Top-P, and Repetition Penalties with live probability distributions.
5. **OpenAI HTTP API & SSE Stream**: Real-time Server-Sent Events terminal simulator.
6. **Prompt Lookup Decoding**: Zero-model N-Gram suffix matcher with live context extraction.
7. **Cargo Workspace Architecture**: Interactive breakdown of crates, tests, and publishing workflow.
---
## 📦 Publishing to Crates.io
Publish from the bottom of the dependency tree upward:
```bash
cargo publish -p spec-stream-core
cargo publish -p spec-stream-cache
cargo publish -p spec-stream-engine
cargo publish -p spec-stream
cargo publish -p spec-stream-server
```
---
## 📄 License
Dual-licensed under MIT and Apache-2.0.