Struct kalosm_language_model::TokenOutputStream
source · pub struct TokenOutputStream { /* private fields */ }
Expand description
This is a wrapper around a tokenizer to ensure that tokens can be returned to the user in a streaming way rather than having to wait for the full decoding.
Implementations§
source§impl TokenOutputStream
impl TokenOutputStream
sourcepub fn new(
tokenizer: Arc<dyn Tokenizer + Send + Sync>,
tokens: impl Into<Vec<u32>>
) -> Self
pub fn new( tokenizer: Arc<dyn Tokenizer + Send + Sync>, tokens: impl Into<Vec<u32>> ) -> Self
Creates a new token output stream.
sourcepub fn sample_token(
&self,
sampler: &mut impl Sampler,
logits: Logits,
stop_on: Option<&str>
) -> Result<u32>
pub fn sample_token( &self, sampler: &mut impl Sampler, logits: Logits, stop_on: Option<&str> ) -> Result<u32>
Samples a token from the logits.
sourcepub fn decode_rest(&self) -> Result<Option<String>>
pub fn decode_rest(&self) -> Result<Option<String>>
Decode the remaining tokens.
sourcepub fn decode_all(&self) -> Result<String>
pub fn decode_all(&self) -> Result<String>
Decode all tokens.
Auto Trait Implementations§
impl !RefUnwindSafe for TokenOutputStream
impl Send for TokenOutputStream
impl Sync for TokenOutputStream
impl Unpin for TokenOutputStream
impl !UnwindSafe for TokenOutputStream
Blanket Implementations§
source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more