Skip to main content

Crate spec_stream_core

Crate spec_stream_core 

Source
Expand description

§spec-stream-core

Core primitives, data structures, and mathematical verification algorithms for speculative decoding in high-throughput Large Language Model (LLM) inference.

Designed to be model-agnostic, zero-allocation where possible, and strictly compliant with exact output probability distributions (Leviathan et al., 2023).

Re-exports§

pub use error::CoreError;
pub use logits::Logits;
pub use logits::TokenId;
pub use metrics::SpeculativeMetrics;
pub use tokenizer::ByteLevelTokenizer;
pub use tokenizer::Tokenizer;
pub use tree::SpeculativeTree;
pub use tree::TreeNode;
pub use tree::TreeVerificationResult;
pub use types::DraftProposal;
pub use types::SamplingParams;
pub use types::TokenCandidate;
pub use types::VerificationResult;
pub use types::VerificationStrategy;
pub use verifier::SpeculativeVerifier;

Modules§

error
logits
metrics
tokenizer
Tokenizer
tree
types
verifier