Struct USif

pub struct USif<'w, 'p, W, P> { /* private fields */ }

An implementation of uSIF.

uSIF is Unsupervised Smooth Inverse Frequency and Piecewise Common Component Removal, simple but pewerful techniques for sentence embeddings described in the paper: Kawin Ethayarajh, Unsupervised Random Walk Sentence Embeddings: A Strong but Simple Baseline, RepL4NLP 2018.

Brief description of API

The algorithm consists of two steps:

1. Compute sentence embeddings with the uSIF weighting.
2. Remove the common components from the sentence embeddings.

The weighting parameter and common components are computed from input sentences.

Our API is designed to allow reuse of these values once computed because it is not always possible to obtain a sufficient number of sentences as queries to compute.

USif::fit computes these values from input sentences and returns a fitted instance of USif. USif::embeddings computes sentence embeddings with the fitted values.

Examples

use std::io::BufReader;

use finalfusion::compat::text::ReadText;
use finalfusion::embeddings::Embeddings;
use wordfreq::WordFreq;

use sif_embedding::{USif, SentenceEmbedder};

// Loads word embeddings from a pretrained model.
let word_embeddings_text = "las 0.0 1.0 2.0\nvegas -3.0 -4.0 -5.0\n";
let mut reader = BufReader::new(word_embeddings_text.as_bytes());
let word_embeddings = Embeddings::read_text(&mut reader)?;

// Loads word probabilities from a pretrained model.
let word_probs = WordFreq::new([("las", 0.4), ("vegas", 0.6)]);

// Prepares input sentences.
let sentences = ["las vegas", "mega vegas"];

// Fits the model with input sentences.
let model = USif::new(&word_embeddings, &word_probs);
let model = model.fit(&sentences)?;

// Computes sentence embeddings in shape (n, m),
// where n is the number of sentences and m is the number of dimensions.
let sent_embeddings = model.embeddings(sentences)?;
assert_eq!(sent_embeddings.shape(), &[2, 3]);

Only uSIF weighting

If you want to apply only the uSIF weighting to avoid the computation of common components, use USif::with_parameters and set n_components to 0.

use std::io::BufReader;

use finalfusion::compat::text::ReadText;
use finalfusion::embeddings::Embeddings;
use wordfreq::WordFreq;

use sif_embedding::{USif, SentenceEmbedder};

// Loads word embeddings from a pretrained model.
let word_embeddings_text = "las 0.0 1.0 2.0\nvegas -3.0 -4.0 -5.0\n";
let mut reader = BufReader::new(word_embeddings_text.as_bytes());
let word_embeddings = Embeddings::read_text(&mut reader)?;

// Loads word probabilities from a pretrained model.
let word_probs = WordFreq::new([("las", 0.4), ("vegas", 0.6)]);

// Prepares input sentences.
let sentences = ["las vegas", "mega vegas"];

// When setting `n_components` to `0`, no common components are removed.
let model = USif::with_parameters(&word_embeddings, &word_probs, 0);
let model = model.fit(&sentences)?;
let sent_embeddings = model.embeddings(sentences)?;
assert_eq!(sent_embeddings.shape(), &[2, 3]);

Serialization of fitted parameters

If you want to serialize and deserialize the fitted parameters, use USif::serialize and USif::deserialize.

use std::io::BufReader;

use approx::assert_relative_eq;
use finalfusion::compat::text::ReadText;
use finalfusion::embeddings::Embeddings;
use wordfreq::WordFreq;

use sif_embedding::{USif, SentenceEmbedder};

// Loads word embeddings from a pretrained model.
let word_embeddings_text = "las 0.0 1.0 2.0\nvegas -3.0 -4.0 -5.0\n";
let mut reader = BufReader::new(word_embeddings_text.as_bytes());
let word_embeddings = Embeddings::read_text(&mut reader)?;

// Loads word probabilities from a pretrained model.
let word_probs = WordFreq::new([("las", 0.4), ("vegas", 0.6)]);

// Prepares input sentences.
let sentences = ["las vegas", "mega vegas"];

// Fits the model and computes sentence embeddings.
let model = USif::new(&word_embeddings, &word_probs);
let model = model.fit(&sentences)?;
let sent_embeddings = model.embeddings(&sentences)?;

// Serializes and deserializes the fitted parameters.
let bytes = model.serialize()?;
let other = USif::deserialize(&bytes, &word_embeddings, &word_probs)?;
let other_embeddings = other.embeddings(&sentences)?;
assert_relative_eq!(sent_embeddings, other_embeddings);

Implementations

impl<'w, 'p, W, P> USif<'w, 'p, W, P>

where W: WordEmbeddings, P: WordProbabilities,

pub const fn new(word_embeddings: &'w W, word_probs: &'p P) -> Self

Creates a new instance with default parameters defined by DEFAULT_N_COMPONENTS.

Arguments

• word_embeddings - Word embeddings.
• word_probs - Word probabilities.

pub const fn with_parameters( word_embeddings: &'w W, word_probs: &'p P, n_components: usize, ) -> Self

Creates a new instance with manually specified parameters.

Arguments

• word_embeddings - Word embeddings.
• word_probs - Word probabilities.
• n_components - The number of principal components to remove.

When setting n_components to 0, no principal components are removed.

pub const fn separator(self, separator: char) -> Self

Sets a separator for sentence segmentation (default: DEFAULT_SEPARATOR).

pub fn n_samples_to_fit(self, n_samples_to_fit: usize) -> Result<Self>

Sets the number of samples to fit the model (default: DEFAULT_N_SAMPLES_TO_FIT).

Errors

Returns an error if n_samples_to_fit is 0.

pub fn serialize(&self) -> Result<Vec<u8>>

Serializes the model.

pub fn deserialize( bytes: &[u8], word_embeddings: &'w W, word_probs: &'p P, ) -> Result<Self>

Deserializes the model.

Arguments

• bytes - Byte sequence exported by Self::serialize.
• word_embeddings - Word embeddings.
• word_probs - Word probabilities.

word_embeddings and word_probs must be the same as those used in serialization.

Trait Implementations

impl<'w, 'p, W: Clone, P: Clone> Clone for USif<'w, 'p, W, P>

fn clone(&self) -> USif<'w, 'p, W, P>

Returns a duplicate of the value.

const fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<W, P> SentenceEmbedder for USif<'_, '_, W, P>

where W: WordEmbeddings, P: WordProbabilities,

fn embedding_size(&self) -> usize

Returns the number of dimensions for sentence embeddings, which is the same as the number of dimensions for word embeddings.

fn fit<S>(self, sentences: &[S]) -> Result<Self>

where S: AsRef<str>,

Fits the model with input sentences.

Sentences to fit are randomly sampled from sentences with Self::n_samples_to_fit.

Errors

Returns an error if sentences is empty.

fn embeddings<I, S>(&self, sentences: I) -> Result<Array2<Float>>

where I: IntoIterator<Item = S>, S: AsRef<str>,

Computes embeddings for input sentences using the fitted model.

Errors

Returns an error if the model is not fitted.