Struct PrependableEncoder

pub struct PrependableEncoder<'a> { /* private fields */ }

Encoder which keeps track of the encoding length while prepending characters.

Implementations

impl<'a> PrependableEncoder<'a>

pub fn new(bpe: &'a BytePairEncoding) -> PrependableEncoder<'a>

pub fn truncate(&mut self, len: usize)

pub fn extend(&mut self, iter: impl Iterator<Item = u8>)

Prepends multiple bytes to the input string.

pub fn push(&mut self, c: u8)

Prepends a byte to the input string which should be tokenized. The operation is amortized O(1) (due to vector resizing).

pub fn token_count(&self) -> usize

Returns the number of tokens required to tokenize the input text. This operation is O(1) and can be called at any point in time.

pub fn len(&self) -> usize

pub fn is_empty(&self) -> bool

Returns true if the structure represents the empty string.

Trait Implementations

impl<'a> Clone for PrependableEncoder<'a>

fn clone(&self) -> PrependableEncoder<'a>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.