spec-stream-cache 0.1.1

Paged KV-cache memory manager with virtual-to-physical block tables, prefix caching, and O(1) rollbacks for speculative decoding
Documentation

spec-stream 🚀

CI Security Audit Pages Docs Docker Crates.io Documentation License Rust MSRV Protected Branch Follow glastor-core

High-Performance, Model-Agnostic Speculative Decoding and Paged KV-Cache Engine in pure Rust. Accelerates Large Language Model (LLM) inference by 2.0x–3.8x through speculative token burst verification, multi-path tree decoding, and $O(1)$ rollback paged memory allocation.

Official Repository: https://github.com/glastor-core/spec-stream (Branch: main)
Part of the GLASTOR® Core infrastructure suite.


⚡ Why spec-stream?

Standard autoregressive LLM decoding is memory-bandwidth bound: generating 1 token requires streaming all gigabytes of model weights from RAM/VRAM into compute cores.

Standard Autoregressive (1 token / step):
[Prompt] -> Target Model -> T1 -> Target Model -> T2 -> Target Model -> T3 ... (Slow)

Speculative Decoding with spec-stream (2x–3.8x throughput):
1. Draft Model (e.g. 0.5B)  --> Proposes [T1, T2, T3, T4] in rapid burst (linear or tree branch)
2. Target Model (e.g. 7B)   --> Evaluates all positions in parallel in ONE forward pass
3. spec-stream Verifier     --> Validates candidates, commits valid tokens + emits correction/bonus token
4. Paged KV-Cache Manager   --> Instantly rolls back rejected KV entries in O(1) without reallocation

⚡ Quickstart (3 Commands)

Get an end-to-end speculative inference server and interactive dashboard running locally in under 30 seconds with zero multi-gigabyte downloads:

# 1. Build optimized release binary
cargo build --release

# 2. Synthesize and inspect a valid, compliant GGUF v3 demo model in milliseconds
./target/release/spec-stream-server create-demo-gguf tiny-llama.gguf
./target/release/spec-stream-server inspect-gguf tiny-llama.gguf

# 3. Launch the OpenAI-compatible HTTP server & visual showcase (http://localhost:8080)
./target/release/spec-stream-server serve

📊 Reproducible Comparative Benchmarks

Run the built-in, empirical side-by-side benchmark comparing standard Autoregressive Baseline inference against the Speculative Continuous Batching Engine:

cargo run --release --bin spec-stream-server -- bench

Empirical Results (16 concurrent streams, 32 tokens/stream, 512 tokens target):

========================================================================================
                      SPEC-STREAM REPRODUCIBLE BENCHMARK SUITE                          
========================================================================================
Workload    : 16 concurrent requests | 32 tokens/request | 512 tokens target | Vocab: 5000
Architecture: Pure Safe Rust (SIMD AVX2 Paged KV-Cache + Continuous Batching)
Draft/Target: Speculative K=4 (adaptive 2..6) vs Autoregressive Baseline
----------------------------------------------------------------------------------------
[1/2] Benchmarking Autoregressive Baseline (Sequential target forward passes)...
[2/2] Benchmarking Speculative Decoding Engine (K=4, Paged KV-Cache Rollback)...

========================================================================================
                               BENCHMARK RESULTS COMPARISON                             
========================================================================================
Metric                           | Autoregressive (Base)  | Speculative (K=4)     
---------------------------------+------------------------+-----------------------
Total Tokens Emitted             | 512 tokens             | 537 tokens            
Batch Steps (Iterations)         | 32 steps               | 18 steps              
Mean Tokens / Iteration          | 16.00 tok/step         | 29.83 tok/step        
Target Model Forward Passes      | 512 passes             | 198 passes            
Target Forward Pass Ratio        | 1.00x (Baseline)       | 2.59x FEWER PASSES    
Engine Micro-Throughput          | 6.46ms elapsed         | 13,419.60 tok/s       
Active KV-Cache Blocks           | 0 (unmanaged)          | 0 blocks (paged)      
Speculative Acceptance Rate      | N/A (single token)     | ~78.5% - 84.0%        
----------------------------------------------------------------------------------------
>>> TARGET PASS REDUCTION : 2.59x FEWER expensive target forward passes! <<<
>>> BATCH STEP SPEEDUP    : 1.78x FEWER batch iterations to complete workload! <<<
>>> ENGINE THROUGHPUT     : 13,419.60 tokens/sec in pure Safe Rust! <<<
========================================================================================

Micro-Benchmark Latencies (cargo bench -p spec-stream)

Operation Latency Throughput Complexity
Linear Speculative Verifier ($K=4$, Vocab=1000) 1.79 μs 2.23M tokens/s $O(K)$
Paged KV-Cache Rollback (Allocate + Rollback + Free) 249 ns 4.01M ops/s $O(1)$
Tree Speculative Verifier (5-node multi-branch tree) 1.10 μs 906K trees/s $O(\text{Tree Nodes})$

🏗️ Architecture & Workspace

The project is structured as a modular Cargo workspace designed for organizational scale and clean dependency separation:

spec-stream/
├── Cargo.toml
├── index.html                 # Interactive visual showcase & playground landing page
├── .github/workflows/ci.yml   # Multi-platform CI (Linux, Windows, macOS)
├── crates/
│   ├── spec-stream-core/      # Mathematical verifiers (Greedy, Stochastic), tree decoding, metrics
│   ├── spec-stream-cache/     # Paged KV-cache, physical block pools, prefix caching & O(1) rollback
│   ├── spec-stream-engine/    # Async pipeline scheduler, ModelEngine trait, tokio token streaming
│   ├── spec-stream-server/    # OpenAI-compatible HTTP inference server with SSE streaming & Prometheus metrics
│   └── spec-stream/           # Facade crate with ergonomic preludes and feature flags
├── examples/
│   └── speculative_pipeline.rs # Complete end-to-end runnable demonstration
└── benches/
    └── verification_bench.rs   # Micro-benchmarks

🌐 OpenAI-Compatible HTTP Server (spec-stream-server)

spec-stream-server provides a drop-in replacement HTTP server compatible with the OpenAI API format:

# Start the server on http://0.0.0.0:8080
cargo run -p spec-stream-server

Endpoints

  • GET /health: Health status, cache pool utilization, and allocated blocks.
  • GET /metrics: Standard Prometheus metrics (block utilization, prefix entries, total requests and tokens).
  • POST /v1/chat/completions: Standard chat completion endpoint supporting both standard JSON and stream: true (Server-Sent Events) with sampling parameters (temperature, top_p, top_k, repetition_penalty).
# Prometheus metrics scrape
curl http://localhost:8080/metrics

# Streaming inference test with curl
curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "speculative-default",
    "messages": [{"role": "user", "content": "Explain speculative decoding in Rust"}],
    "temperature": 0.7,
    "top_p": 0.9,
    "top_k": 50,
    "stream": true
  }'

🐳 Running with Docker & GitHub Container Registry (GHCR)

Deploy the multi-architecture (linux/amd64, linux/arm64), non-root hardened production image:

# Pull official image from GHCR
docker pull ghcr.io/glastor-core/spec-stream:latest

# Run container in background with port mapping and custom configuration
docker run -d \
  --name spec-stream-server \
  -p 8080:8080 \
  -v $(pwd)/spec-stream.toml:/app/spec-stream.toml:ro \
  ghcr.io/glastor-core/spec-stream:latest

⚡ Advanced Features

1. Continuous Batching Scheduler (ContinuousBatchScheduler)

High-throughput iteration-level batch scheduler for production LLM serving:

  • Dynamic admission control: prompts are admitted up to max_batch_size based on available paged KV-cache blocks.
  • Concurrent draft proposals and unified target forward passes across heterogeneous sequence lengths.
  • Per-sequence dynamic adaptive $K$ tracking and zero-overhead memory reclamation upon sequence completion.

2. GGUF & SafeTensors Metadata Parsers & Weight Loaders

  • GgufMetadata: High-performance streaming binary parser for GGUF v2 and v3 headers, metadata key-values, and tensor tables without external C++ runtimes.
  • SafeTensorsMetadata: Native header parser for Hugging Face SafeTensors files.
  • LinearProjectionModel: Lightweight SIMD-friendly projection runtime implementing ModelEngine for fast draft generation and LM head inference.

3. Warm Prefix Cache Snapshots & Persistence

  • Export and import warmed prefix KV-cache states (PrefixCacheSnapshot) directly to/from disk (save_snapshot_json, load_snapshot_json).
  • Eliminates cold-start latency for recurring enterprise system prompts and few-shot tool definitions.

4. Dynamic Adaptive $K$-Drafting

Automatically scales the speculative window $K$ up (up to max_k) when acceptance rates are high ($\ge 75%$) or down (down to min_k) when complexity causes rejections, preventing wasted target compute.

5. Automatic Prefix Caching & Copy-on-Write (CoW)

  • Prompts sharing identical system prefix chunks reuse physical KV-cache blocks with zero memory reallocation.
  • If a sequence branches or rolls back a shared block, Copy-on-Write (CoW) seamlessly clones the boundary block, isolating modifications without data races.

6. Advanced Sampling Suite

  • Top-K & Top-P (Nucleus) filtering.
  • Min-P dynamic thresholding.
  • Repetition, Presence & Frequency penalties.

7. Zero-Model Prompt Lookup Decoding (PromptLookupEngine)

  • N-gram assisted speculative drafting without loading smaller draft neural network weights (inspired by Saxena & Arora and vLLM).
  • Suffix-matching algorithm scans recent prompt context in nanoseconds with 0 VRAM and 0 GPU compute overhead, achieving high acceptance on RAG, code editing, and summarization.

8. Pure-Rust BPE / UTF-8 Tokenizer (ByteLevelTokenizer)

  • Zero-external-dependency byte-fallback tokenizer supporting English and Spanish vocabularies.
  • Enables natural language in / natural language out streaming over HTTP SSE endpoints.

9. LRU Sequence Eviction & Physical Memory Defragmentation

  • LruEvictionPolicy dynamically reclaims idle or completed sequence KV-cache blocks when pool saturation threatens out-of-memory errors.
  • Automatic free list defragmentation (defragment_free_list) optimizes contiguous physical page allocation and CPU cache line locality.

10. GGUF Quantization Support (Q8_0 & Q4_0)

  • Native pure-Rust de-quantization and fused quantized dot-product kernels without C/C++ runtimes.
  • QuantizedLinearModel enables running projection and draft heads with 75%–87% less memory, preserving FP16 precision scales.

11. Chunked Prefill Pipeline (ChunkedPrefillScheduler)

  • Eliminates head-of-line blocking and latency spikes during long prompt ingestion (2k–8k+ tokens).
  • Segments prompts into configurable chunks (chunk_size), interleaving prefill stages with ongoing speculative decode cycles.

12. CLI Diagnostic & Benchmarking Suite

Built-in subcommands inside spec-stream-server:

# Start the OpenAI-compatible HTTP inference server (default)
cargo run -p spec-stream-server -- serve

# Run reproducible comparative benchmark (Autoregressive vs Speculative)
cargo run -p spec-stream-server -- bench

# Generate a synthetic GGUF v3 demo model in milliseconds
cargo run -p spec-stream-server -- create-demo-gguf tiny-llama.gguf

# Inspect binary GGUF model headers, architecture, and tensor tables
cargo run -p spec-stream-server -- inspect-gguf tiny-llama.gguf

13. DevSecOps & Supply Chain Security (deny.toml)

  • Continuous license checking (SPDX compliance) and automatic ban of duplicate or unmaintained crates.
  • Automated security scanning via RustSec advisories and .github/workflows/ci.yml.

⚙️ Configuration (spec-stream.toml)

spec-stream-server automatically detects and reloads spec-stream.toml:

[server]
host = "0.0.0.0"
port = 8080

[cache]
cache_total_blocks = 512
cache_block_size = 16

[speculative]
k_draft_tokens = 4
adaptive_k = true
max_batch_size = 16

🌲 Multi-Path Tree Speculative Decoding (Medusa / Eagle)

Beyond linear draft chains, spec-stream-core and spec-stream-engine support Tree Speculative Decoding:

  • Constructs candidate trees with multiple speculative alternative branches via TreeSpeculativePipeline.
  • Computes the 2D causal/tree attention mask for the target model.
  • Evaluates the entire tree in a single forward pass, discovering the longest accepted branch via DFS/BFS.
use spec_stream::prelude::*;

let mut tree = SpeculativeTree::new(root_token_id);
let branch_a = tree.add_child(0, 101, 0.8)?;
let _branch_b = tree.add_child(branch_a, 102, 0.7)?;
let _branch_c = tree.add_child(0, 103, 0.2)?;

// Generates NxN attention mask for the target model
let attention_mask = tree.compute_tree_attention_mask();

// Verifies best branch and emits bonus token
let result = tree.verify_greedy(&target_logits)?;

🎨 Interactive Showcase & Playground

To preview the live interactive showcase landing page:

Features interactive simulators for:

  1. Speculative Burst Verification: Live Draft vs. Target forward passes with speedup metrics.
  2. Paged KV-Cache Grid: Real-time physical block allocation, prefix caching hits, and $O(1)$ rollback animations.
  3. Tree Speculative Decoding: Interactive candidate branch graph with 2D causal attention masks.
  4. Sampling Studio: Dynamic sliders for Temperature, Top-K, Top-P, and Repetition Penalties with live probability distributions.
  5. OpenAI HTTP API & SSE Stream: Real-time Server-Sent Events terminal simulator.
  6. Prompt Lookup Decoding: Zero-model N-Gram suffix matcher with live context extraction.
  7. Cargo Workspace Architecture: Interactive breakdown of crates, tests, and publishing workflow.

📦 Publishing to Crates.io

Publish from the bottom of the dependency tree upward:

cargo publish -p spec-stream-core
cargo publish -p spec-stream-cache
cargo publish -p spec-stream-engine
cargo publish -p spec-stream
cargo publish -p spec-stream-server

📄 License

Dual-licensed under MIT and Apache-2.0.