Skip to main content

Crate spec_stream

Crate spec_stream 

Source
Expand description

§spec-stream

spec-stream is an ultra-fast, modular, model-agnostic speculative decoding and paged KV-cache engine in pure Rust.

Accelerates Large Language Model (LLM) inference by 2x–3x through speculative token burst verification and $O(1)$ rollback paged memory allocation.

§Key Capabilities

  • Mathematical Exactness: Implements Leviathan et al. (2023) stochastic rejection sampling, guaranteeing output probability distribution identical to the target model.
  • Paged KV-Cache with O(1) Rollback: Physical-to-virtual block tables prevent memory fragmentation and allow instant rewind of rejected speculative proposals.
  • Model Agnostic: Plug in any inference backend: Candle, Burn, ONNX Runtime, or native C/CUDA FFI.
  • Async Token Streaming: First-class support for Tokio streams and Server-Sent Events (SSE).

Re-exports§

pub use spec_stream_core as core;
pub use spec_stream_cache as cache;
pub use spec_stream_engine as engine;

Modules§

prelude
Convenient prelude re-exporting core traits and common structs.