Struct sif_embedding::sif::Sif

pub struct Sif<'w, 'p, W, P> { /* private fields */ }

An implementation of Smooth Inverse Frequency and Common Component Removal, simple but pewerful techniques for sentence embeddings described in the paper: Sanjeev Arora, Yingyu Liang, and Tengyu Ma, A Simple but Tough-to-Beat Baseline for Sentence Embeddings, ICLR 2017.

Brief description of API

The algorithm consists of two steps:

1. Compute sentence embeddings with the SIF weighting.
2. Remove the common components from the sentence embeddings.

The common components are computed from input sentences.

Our API is designed to allow reuse of common components once computed because it is not always possible to obtain a sufficient number of sentences as queries to compute.

Sif::fit computes the common components from input sentences and returns a fitted instance of Sif. Sif::embeddings computes sentence embeddings with the fitted components.

If you find these two steps annoying, you can use Sif::fit_embeddings.

Examples

use std::io::BufReader;

use finalfusion::compat::text::ReadText;
use finalfusion::embeddings::Embeddings;
use wordfreq::WordFreq;

use sif_embedding::{Sif, SentenceEmbedder};

// Loads word embeddings from a pretrained model.
let word_embeddings_text = "las 0.0 1.0 2.0\nvegas -3.0 -4.0 -5.0\n";
let mut reader = BufReader::new(word_embeddings_text.as_bytes());
let word_embeddings = Embeddings::read_text(&mut reader)?;

// Loads word probabilities from a pretrained model.
let word_probs = WordFreq::new([("las", 0.4), ("vegas", 0.6)]);

// Computes sentence embeddings in shape (n, m),
// where n is the number of sentences and m is the number of dimensions.
let model = Sif::new(&word_embeddings, &word_probs);
let (sent_embeddings, model) = model.fit_embeddings(&["las vegas", "mega vegas"])?;
assert_eq!(sent_embeddings.shape(), &[2, 3]);

// Once fitted, the parameters can be used to compute sentence embeddings.
let sent_embeddings = model.embeddings(["vegas pro"])?;
assert_eq!(sent_embeddings.shape(), &[1, 3]);

Only SIF weighting

If you want to apply only the SIF weighting to avoid the computation of common components, use Sif::with_parameters and set n_components to 0. In this case, you can skip Sif::fit and directly perform Sif::embeddings because there is no parameter to fit (although the quality of the embeddings may be worse).

use std::io::BufReader;

use finalfusion::compat::text::ReadText;
use finalfusion::embeddings::Embeddings;
use wordfreq::WordFreq;

use sif_embedding::{Sif, SentenceEmbedder};

// Loads word embeddings from a pretrained model.
let word_embeddings_text = "las 0.0 1.0 2.0\nvegas -3.0 -4.0 -5.0\n";
let mut reader = BufReader::new(word_embeddings_text.as_bytes());
let word_embeddings = Embeddings::read_text(&mut reader)?;

// Loads word probabilities from a pretrained model.
let word_probs = WordFreq::new([("las", 0.4), ("vegas", 0.6)]);

// When setting `n_components` to `0`, no common components are removed, and
// the sentence embeddings can be computed without `fit`.
let model = Sif::with_parameters(&word_embeddings, &word_probs, 1e-3, 0)?;
let sent_embeddings = model.embeddings(["las vegas", "mega vegas"])?;
assert_eq!(sent_embeddings.shape(), &[2, 3]);

Serialization of fitted parameters

If you want to serialize and deserialize the fitted parameters, use Sif::serialize and Sif::deserialize.

use std::io::BufReader;

use approx::assert_relative_eq;
use finalfusion::compat::text::ReadText;
use finalfusion::embeddings::Embeddings;
use wordfreq::WordFreq;

use sif_embedding::{Sif, SentenceEmbedder};

// Loads word embeddings from a pretrained model.
let word_embeddings_text = "las 0.0 1.0 2.0\nvegas -3.0 -4.0 -5.0\n";
let mut reader = BufReader::new(word_embeddings_text.as_bytes());
let word_embeddings = Embeddings::read_text(&mut reader)?;

// Loads word probabilities from a pretrained model.
let word_probs = WordFreq::new([("las", 0.4), ("vegas", 0.6)]);

// Computes sentence embeddings in shape (n, m),
// where n is the number of sentences and m is the number of dimensions.
let model = Sif::new(&word_embeddings, &word_probs);
let (sent_embeddings, model) = model.fit_embeddings(&["las vegas", "mega vegas"])?;

// Serializes and deserializes the fitted parameters.
let bytes = model.serialize()?;
let other = Sif::deserialize(&bytes, &word_embeddings, &word_probs)?;
let other_embeddings = other.embeddings(["las vegas", "mega vegas"])?;
assert_relative_eq!(sent_embeddings, other_embeddings);

Implementations

impl<'w, 'p, W, P> Sif<'w, 'p, W, P>where W: WordEmbeddings, P: WordProbabilities,

pub const fn new(word_embeddings: &'w W, word_probs: &'p P) -> Self

Creates a new instance with default parameters defined by DEFAULT_PARAM_A and DEFAULT_N_COMPONENTS.

Arguments

• word_embeddings - Word embeddings.
• word_probs - Word probabilities.

pub fn with_parameters( word_embeddings: &'w W, word_probs: &'p P, param_a: Float, n_components: usize ) -> Result<Self>

Creates a new instance with manually specified parameters.

Arguments

• word_embeddings - Word embeddings.
• word_probs - Word probabilities.
• param_a - A parameter a for SIF-weighting that should be positive.
• n_components - The number of principal components to remove.

When setting n_components to 0, no principal components are removed.

Errors

Returns an error if param_a is not positive.

pub const fn separator(self, separator: char) -> Self

Sets a separator for sentence segmentation (default: DEFAULT_SEPARATOR).

pub fn serialize(&self) -> Result<Vec<u8>>

Serializes the model.

pub fn deserialize( bytes: &[u8], word_embeddings: &'w W, word_probs: &'p P ) -> Result<Self>

Deserializes the model.

Arguments

• bytes - Byte sequence exported by Self::serialize.
• word_embeddings - Word embeddings.
• word_probs - Word probabilities.

word_embeddings and word_probs must be the same as those used in serialization.

Trait Implementations

impl<'w, 'p, W: Clone, P: Clone> Clone for Sif<'w, 'p, W, P>

fn clone(&self) -> Sif<'w, 'p, W, P>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'w, 'p, W, P> SentenceEmbedder for Sif<'w, 'p, W, P>where W: WordEmbeddings, P: WordProbabilities,

fn embedding_size(&self) -> usize

Returns the number of dimensions for sentence embeddings, which is the same as the number of dimensions for word embeddings.

fn fit<S>(self, sentences: &[S]) -> Result<Self>where S: AsRef<str>,

Fits the model with input sentences.

Errors

Returns an error if sentences is empty.

Notes

If n_components is 0, does nothing and returns self.

fn embeddings<I, S>(&self, sentences: I) -> Result<Array2<Float>>where I: IntoIterator<Item = S>, S: AsRef<str>,

Computes embeddings for input sentences using the fitted model.

Errors

Returns an error if the model is not fitted.

Notes

If n_components is 0, the fitting is not required.

fn fit_embeddings<S>(self, sentences: &[S]) -> Result<(Array2<Float>, Self)>where S: AsRef<str>,

Fits the model with input sentences and computes embeddings using it, providing the same behavior as performing Self::fit and then Self::embeddings.