pub struct Speculation<'a> {
pub draft_tokens: &'a [TokenId],
pub draft_logits: &'a [Vec<f32>],
pub target_logits: &'a [Vec<f32>],
pub temperature: f32,
}Expand description
Input to verify_speculation: logit vectors at each speculation position.
For a speculation of N draft tokens:
draft_logits.len() == N— the distribution the draft model used to sample each draft token (aligned withdraft_tokens).target_logits.len() == N + 1— the target model’s distributions at each position; the extra slot is the bonus-token draw after all drafts are accepted.draft_tokens.len() == N.
Fields§
§draft_tokens: &'a [TokenId]§draft_logits: &'a [Vec<f32>]§target_logits: &'a [Vec<f32>]§temperature: f32Auto Trait Implementations§
impl<'a> Freeze for Speculation<'a>
impl<'a> RefUnwindSafe for Speculation<'a>
impl<'a> Send for Speculation<'a>
impl<'a> Sync for Speculation<'a>
impl<'a> Unpin for Speculation<'a>
impl<'a> UnsafeUnpin for Speculation<'a>
impl<'a> UnwindSafe for Speculation<'a>
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
impl<T> ErasedDestructor for Twhere
T: 'static,
Source§impl<T> Instrument for T
impl<T> Instrument for T
Source§fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
Source§fn in_current_span(self) -> Instrumented<Self> ⓘ
fn in_current_span(self) -> Instrumented<Self> ⓘ
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more