spec-stream 0.1.0

High-performance, model-agnostic speculative decoding and paged KV-cache engine for accelerated LLM inference
Documentation
//! # spec-stream
//!
//! **spec-stream** is an ultra-fast, modular, model-agnostic speculative decoding and
//! paged KV-cache engine in pure Rust.
//!
//! Accelerates Large Language Model (LLM) inference by 2x–3x through speculative token burst
//! verification and $O(1)$ rollback paged memory allocation.
//!
//! ## Key Capabilities
//! - **Mathematical Exactness**: Implements Leviathan et al. (2023) stochastic rejection sampling,
//!   guaranteeing output probability distribution identical to the target model.
//! - **Paged KV-Cache with O(1) Rollback**: Physical-to-virtual block tables prevent memory fragmentation
//!   and allow instant rewind of rejected speculative proposals.
//! - **Model Agnostic**: Plug in any inference backend: Candle, Burn, ONNX Runtime, or native C/CUDA FFI.
//! - **Async Token Streaming**: First-class support for Tokio streams and Server-Sent Events (SSE).

pub use spec_stream_core as core;

#[cfg(feature = "cache")]
pub use spec_stream_cache as cache;

#[cfg(feature = "engine")]
pub use spec_stream_engine as engine;

/// Convenient prelude re-exporting core traits and common structs.
pub mod prelude {
    pub use spec_stream_core::{
        ByteLevelTokenizer, DraftProposal, Logits, SamplingParams, SpeculativeMetrics,
        SpeculativeTree, SpeculativeVerifier, TokenCandidate, TokenId, Tokenizer,
        TreeVerificationResult, VerificationResult, VerificationStrategy,
    };

    #[cfg(feature = "cache")]
    pub use spec_stream_cache::{
        BlockAllocator, BlockTable, EvictionPolicy, LruEvictionPolicy, PagedKvCacheManager,
        PhysicalBlock, PhysicalBlockId, PrefixCacheSnapshot, PrefixSnapshotEntry,
    };

    #[cfg(feature = "engine")]
    pub use spec_stream_engine::{
        ChunkedPrefillConfig, ChunkedPrefillScheduler, ContinuousBatchScheduler, GenerationEvent,
        GgufMetadata, LinearProjectionModel, ModelEngine, PromptLookupConfig, PromptLookupEngine,
        QuantType, QuantizedLinearModel, QueuedRequest, SafeTensorsMetadata, SchedulerStats,
        SpeculativeConfig, SpeculativePipeline, SpeculativeStream, SyntheticModel,
        TreeSpeculativeConfig, TreeSpeculativePipeline,
    };
}