pub struct SpeculativeConfig {
pub k_draft_tokens: usize,
pub max_new_tokens: usize,
pub eos_token_id: Option<TokenId>,
pub strategy: VerificationStrategy,
pub adaptive_k: bool,
pub min_k: usize,
pub max_k: usize,
}Expand description
Configuration parameters for speculative decoding.
Fields§
§k_draft_tokens: usizeNumber of speculative candidate tokens the draft model proposes per cycle (typically 3..6).
max_new_tokens: usizeMaximum new tokens to generate before stopping.
eos_token_id: Option<TokenId>Optional End-of-Sequence token ID to terminate generation.
strategy: VerificationStrategyVerification strategy: Greedy or Stochastic with temperature.
adaptive_k: boolWhether to dynamically adjust K draft tokens based on empirical acceptance rate.
min_k: usizeMinimum allowed K when adaptive_k is enabled (default 1).
max_k: usizeMaximum allowed K when adaptive_k is enabled (default 8).
Trait Implementations§
Source§impl Clone for SpeculativeConfig
impl Clone for SpeculativeConfig
Source§impl Debug for SpeculativeConfig
impl Debug for SpeculativeConfig
Auto Trait Implementations§
impl Forget for SpeculativeConfig
impl Freeze for SpeculativeConfig
impl Move for SpeculativeConfig
impl RefUnwindSafe for SpeculativeConfig
impl Send for SpeculativeConfig
impl Sync for SpeculativeConfig
impl Unpin for SpeculativeConfig
impl UnsafeUnpin for SpeculativeConfig
impl UnwindSafe for SpeculativeConfig
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more