Struct tokenizers::models::unigram::Unigram

pub struct Unigram {
    pub min_score: f64,
    /* private fields */
}

A Unigram model to encode sentences.

Fields

min_score: f64

Implementations

impl Unigram

pub fn from( vocab: Vec<(String, f64)>, unk_id: Option<usize>, byte_fallback: bool ) -> Result<Self>

Create a Unigram model from a given vocabulary. Vocabulary are the various tokens and their associated score which is a sort of a logprob of their frequency, which will enable tokenization and sampling. unk_id, is the index within the vocabulary. For now Unigram requires at least unk because we might find a never seen char. Further versions might allow that part to be hidden.

pub fn byte_fallback(&self) -> bool

pub fn encode(&self, sentence: &str) -> Result<Vec<String>>

This functions take a String, and will encode it in a Vec of Strings, of the best tokenization available to the current model.

use tokenizers::models::unigram::Unigram;

let pieces = vec![
    ("<unk>".to_string(), 0.0),
    ("a".to_string(), 0.0),
    ("b".to_string(), 0.0),
    ("c".to_string(), 0.0),
    ("d".to_string(), 0.0),
    ("cd".to_string(), 1.0),
    ("ab".to_string(), 2.0),
    ("abc".to_string(), 5.0),
    ("abcd".to_string(), 10.0),
];
let model = Unigram::from(pieces, Some(0), false).unwrap();
let result = model.encode("abcdacdxx").unwrap();
assert_eq!(result, vec!["abcd", "a", "cd", "xx"]);

pub fn iter(&self) -> UnigramIterator<'_>

Iterate of vocabulary of the model as a pair of (token, score).

pub fn load<P: AsRef<Path>>(path: P) -> Result<Unigram>

Loads a SentencePiece output model after being trained by tokenizers. After that you can use the model with tokenizers library.

use tokenizers::models::unigram::Unigram;
use std::path::Path;

let model = Unigram::load("mymodel-unigram.json").unwrap();

Trait Implementations

impl Clone for Unigram

fn clone(&self) -> Self

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Unigram

fn fmt(&self, fmt: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Unigram

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for Unigram

fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>

where D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl From<Unigram> for ModelWrapper

fn from(from: Unigram) -> Self

Converts to this type from the input type.

impl Model for Unigram

type Trainer = UnigramTrainer

fn get_vocab(&self) -> HashMap<String, u32>

Retrieve the entire vocabulary mapping (token -> ID)

fn get_vocab_size(&self) -> usize

Retrieve the size of the vocabulary

fn tokenize(&self, sentence: &str) -> Result<Vec<Token>>

Tokenize the given sequence into multiple underlying Token. The offsets on the Token are expected to be relative to the given sequence.

fn token_to_id(&self, token: &str) -> Option<u32>

Find the ID associated to a string token

fn id_to_token(&self, id: u32) -> Option<String>

Find the string token associated to an ID

fn save(&self, folder: &Path, name: Option<&str>) -> Result<Vec<PathBuf>>

Save the current Model in the given folder, using the given prefix for the various files that need to be saved.

fn get_trainer(&self) -> Self::Trainer

Get an instance of a Trainer capable of training this Model

impl PartialEq for Unigram

fn eq(&self, other: &Self) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for Unigram

fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>

where S: Serializer,

Serialize this value into the given Serde serializer.