pub struct Decoder<'a> { /* private fields */ }Expand description
A decoding session: the KV caches, the ids that filled them, and the logits
for whatever comes next. Built by TransformerLm::decoder.
Implementations§
Source§impl<'a> Decoder<'a>
impl<'a> Decoder<'a>
Sourcepub fn fork(&self) -> Result<Decoder<'a>, NetworkError>
pub fn fork(&self) -> Result<Decoder<'a>, NetworkError>
An independent copy of the session, history and caches included. A prefix every request shares, such as a system prompt or a few-shot preamble, is prefilled once and each request continues a fork:
let mut preamble = model.decoder();
preamble.feed(&[1, 2, 3, 4])?;
let mut request = preamble.fork()?;
request.feed(&[5, 6])?;
let id = request.next(&mut sampler)?;On a device the copy takes as much memory as the original’s caches and shares its packed weights.
Sourcepub fn feed(&mut self, ids: &[u32]) -> Result<(), NetworkError>
pub fn feed(&mut self, ids: &[u32]) -> Result<(), NetworkError>
Appends ids to the session: a prompt, or the next turn of a
conversation. Only these tokens are read, not the ones before them.
Sourcepub fn next(&mut self, sampler: &mut Sampler) -> Result<u32, NetworkError>
pub fn next(&mut self, sampler: &mut Sampler) -> Result<u32, NetworkError>
Samples the next token and appends it to the session.
Sourcepub fn logits(&mut self) -> Result<&[f32], NetworkError>
pub fn logits(&mut self) -> Result<&[f32], NetworkError>
The next token’s logits, for choices Sampler does not make:
constrained output, a logit bias, log-probabilities. Append the chosen
id with push:
let mut decoder = model.decoder();
decoder.feed(&[1, 2, 3])?;
let logits = decoder.logits()?;
let id = *allowed.iter().max_by(|a, b| {
logits[**a as usize].total_cmp(&logits[**b as usize])
}).unwrap();
decoder.push(id)?;Auto Trait Implementations§
impl<'a> Freeze for Decoder<'a>
impl<'a> RefUnwindSafe for Decoder<'a>
impl<'a> Send for Decoder<'a>
impl<'a> Sync for Decoder<'a>
impl<'a> Unpin for Decoder<'a>
impl<'a> UnsafeUnpin for Decoder<'a>
impl<'a> UnwindSafe for Decoder<'a>
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more