Struct wordpieces::WordPieces

pub struct WordPieces { /* private fields */ }

A set of word pieces.

Implementations

impl WordPieces

pub fn new(word_initial: Map<Vec<u8>>, continuation: Map<Vec<u8>>) -> Self

Construct new word pieces instance.

The arguments are set of word-initial pieces and the set o continuation pieces. The continuation set pieces must not have continuation markers (such as ##).

pub fn from_buf_read(buf_read: impl BufRead) -> Result<Self, WordPiecesError>

pub fn get_continuation(&self, piece: &str) -> Option<u64>

Look up the index of an initial word piece.

pub fn get_initial(&self, piece: &str) -> Option<u64>

Look up the index of an continuation word piece.

pub fn split<'a, 'b>(&'a self, word: &'b str) -> WordPieceIter<'a, 'b>

Split a string into word pieces.

Returns an iterator over the word pieces.

Trait Implementations

impl From<&WordPieces> for Vec<String>

fn from(word_pieces: &WordPieces) -> Self

Converts to this type from the input type.