Expand description
§spec-stream
spec-stream is an ultra-fast, modular, model-agnostic speculative decoding and paged KV-cache engine in pure Rust.
Accelerates Large Language Model (LLM) inference by 2x–3x through speculative token burst verification and $O(1)$ rollback paged memory allocation.
§Key Capabilities
- Mathematical Exactness: Implements Leviathan et al. (2023) stochastic rejection sampling, guaranteeing output probability distribution identical to the target model.
- Paged KV-Cache with O(1) Rollback: Physical-to-virtual block tables prevent memory fragmentation and allow instant rewind of rejected speculative proposals.
- Model Agnostic: Plug in any inference backend: Candle, Burn, ONNX Runtime, or native C/CUDA FFI.
- Async Token Streaming: First-class support for Tokio streams and Server-Sent Events (SSE).
Re-exports§
pub use spec_stream_core as core;pub use spec_stream_cache as cache;pub use spec_stream_engine as engine;
Modules§
- prelude
- Convenient prelude re-exporting core traits and common structs.