pub struct SpeculativeDecoder<B: Backend> { /* private fields */ }Expand description
Speculative decoder for draft/target model verification.
Implementations§
Source§impl<B: Backend> SpeculativeDecoder<B>
impl<B: Backend> SpeculativeDecoder<B>
Sourcepub fn new(
prediction_config: PredictionConfig,
tree_config: TreeConfig,
max_speculation_length: usize,
) -> Self
pub fn new( prediction_config: PredictionConfig, tree_config: TreeConfig, max_speculation_length: usize, ) -> Self
Create a new speculative decoder.
Sourcepub fn prediction_config(&self) -> &PredictionConfig
pub fn prediction_config(&self) -> &PredictionConfig
Access prediction head configuration.
Sourcepub fn tree_config(&self) -> &TreeConfig
pub fn tree_config(&self) -> &TreeConfig
Access tree configuration.
Sourcepub fn max_speculation_length(&self) -> usize
pub fn max_speculation_length(&self) -> usize
Maximum speculation length.
Sourcepub fn speculate(
&self,
hidden: Tensor<B, 3>,
) -> Result<SpeculativeCandidates, &'static str>
pub fn speculate( &self, hidden: Tensor<B, 3>, ) -> Result<SpeculativeCandidates, &'static str>
Sourcepub fn verify(
&self,
candidates: &SpeculativeCandidates,
target_logits: Tensor<B, 3>,
cache_tokens: Tensor<B, 2>,
) -> Result<SpeculativeVerification<B>, &'static str>
pub fn verify( &self, candidates: &SpeculativeCandidates, target_logits: Tensor<B, 3>, cache_tokens: Tensor<B, 2>, ) -> Result<SpeculativeVerification<B>, &'static str>
Verify candidates against target logits with rejection-style acceptance.
§Shapes
target_logits: [batch, depth, vocab]cache_tokens: [batch, cache_len]
Auto Trait Implementations§
impl<B> Freeze for SpeculativeDecoder<B>
impl<B> RefUnwindSafe for SpeculativeDecoder<B>where
B: RefUnwindSafe,
impl<B> Send for SpeculativeDecoder<B>
impl<B> Sync for SpeculativeDecoder<B>
impl<B> Unpin for SpeculativeDecoder<B>where
B: Unpin,
impl<B> UnwindSafe for SpeculativeDecoder<B>where
B: UnwindSafe,
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more