Struct sentencepiece::SentencePieceProcessor

pub struct SentencePieceProcessor { /* private fields */ }

Sentence piece tokenizer.

Instances of SentencePieceProcessor can be used to tokenizer a sentence using a sentencepiece model.

Implementations

impl SentencePieceProcessor

pub fn from_serialized_proto(data: &[u8]) -> Result<Self, SentencePieceError>

pub fn to_serialized_proto(&self) -> Vec<u8>

Serialize the model to protobuf.

pub fn open(path: impl AsRef<Path>) -> Result<Self, SentencePieceError>

Open a sentencepiece model.

pub fn bos_id(&self) -> Option<u32>

pub fn decode_piece_ids( &self, pieces: &[u32] ) -> Result<String, SentencePieceError>

Decode a sentence from piece identifiers.

pub fn decode_pieces( &self, pieces: &[impl AsRef<str>] ) -> Result<String, SentencePieceError>

pub fn encode( &self, sentence: &str ) -> Result<Vec<PieceWithId>, SentencePieceError>

Encode a sentence as sentence pieces and their identifiers.

pub fn eos_id(&self) -> Option<u32>

pub fn is_empty(&self) -> bool

pub fn len(&self) -> usize

pub fn pad_id(&self) -> Option<u32>

pub fn piece_to_id(&self, piece: &str) -> Result<Option<u32>, NulError>

Get the identifier of a sentence piece.

pub fn sample_encode( &self, sentence: &str, n_best: usize, alpha: f32 ) -> Result<Vec<PieceWithId>, SentencePieceError>

Encode a sentence using sampling (subword regularization).

Sample for the n_best segmentations, where alpha controls the smoothness of the distribution.

This method panics when n_best > 512 or when alpha is not a (normal) positive floating point number.

pub fn unk_id(&self) -> u32

Trait Implementations

impl Debug for SentencePieceProcessor

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Drop for SentencePieceProcessor

fn drop(&mut self)

Executes the destructor for this type.

impl Send for SentencePieceProcessor

impl Sync for SentencePieceProcessor