Struct sentencepiece::SentencePieceProcessor

pub struct SentencePieceProcessor { /* fields omitted */ }

Sentence piece tokenizer.

Instances of SentencePieceProcessor can be used to tokenizer a sentence using a sentencepiece model.

Implementations

impl SentencePieceProcessor

pub fn from_serialized_proto(data: &[u8]) -> Result<Self, SentencePieceError>

pub fn open(path: impl AsRef<Path>) -> Result<Self, SentencePieceError>

Open a sentencepiece model.

pub fn bos_id(&self) -> Option<u32>

pub fn encode(
    &self,
    sentence: &str
) -> Result<Vec<PieceWithId>, SentencePieceError>

Tokenizer a sentence.

pub fn eos_id(&self) -> Option<u32>

pub fn piece_to_id(&self, piece: &str) -> Result<Option<u32>, NulError>

Get the identifier of a sentence piece.

pub fn unknown_id(&self) -> u32

Trait Implementations

impl Debug for SentencePieceProcessor

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Drop for SentencePieceProcessor

fn drop(&mut self)

Executes the destructor for this type.

impl Send for SentencePieceProcessor

impl Sync for SentencePieceProcessor