Struct AppendableEncoder

pub struct AppendableEncoder<'a> { /* private fields */ }

Encoder which keeps track of the encoding length while appending characters.

Implementations

impl<'a> AppendableEncoder<'a>

pub fn new(bpe: &'a BytePairEncoding) -> AppendableEncoder<'a>

pub fn extend(&mut self, iter: impl Iterator<Item = u8>)

Appends multiple bytes to the input string.

pub fn truncate(&mut self, len: usize)

pub fn push(&mut self, c: u8)

Appends a byte to the input string which should be tokenized. The operation is amortized O(1) (due to vector resizing).

pub fn token_count(&self) -> usize

Returns the number of tokens required to tokenize the input text. This operation is O(1) and can be called at any point in time.

pub fn len(&self) -> usize

pub fn is_empty(&self) -> bool

Returns true if the structure represents the empty string.

Trait Implementations

impl<'a> Clone for AppendableEncoder<'a>

fn clone(&self) -> AppendableEncoder<'a>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.