Struct finalfusion::embeddings::Embeddings [−][src]
pub struct Embeddings<V, S> { /* fields omitted */ }
Expand description
Word embeddings.
This data structure stores word embeddings (also known as word vectors) and provides some useful methods on the embeddings, such as similarity and analogy queries.
Implementations
Decompose embeddings in its vocabulary, storage, and optionally norms.
Get metadata mutably.
Set metadata.
Returns the previously-stored metadata.
Get the embedding of a word.
Realize the embedding of a word into the given vector.
This variant of embedding
realizes the embedding into the
given vector. This makes it possible to look up embeddings
without any additional allocations. This method returns
false
and does not modify the vector if no embedding could
be found.
Panics when then the vector does not have the same dimensionality as the word embeddings.
Get a batch of embeddings.
The embeddings of all words
are computed and returned. This method also
return a Vec
indicating for each word if an embedding could be found.
pub fn embedding_batch_into(
&self,
words: &[impl AsRef<str>],
output: ArrayViewMut2<'_, f32>
) -> Vec<bool>
pub fn embedding_batch_into(
&self,
words: &[impl AsRef<str>],
output: ArrayViewMut2<'_, f32>
) -> Vec<bool>
Get a batch of embeddings.
The embeddings of all words
are computed and written to output
. A Vec
is
returned that indicates for each word if an embedding could be found.
This method panics when output
does not have the correct shape.
Get the embedding and original norm of a word.
Returns for a word:
- The word embedding.
- The norm of the embedding before normalization to a unit vector.
The original embedding can be reconstructed by multiplying all embedding components by the original norm.
If the model does not have associated norms, 1 will be returned as the norm for vocabulary words.
Get an iterator over pairs of words and the corresponding embeddings.
pub fn iter_with_norms(&self) -> IterWithNorms<'_>ⓘNotable traits for IterWithNorms<'a>impl<'a> Iterator for IterWithNorms<'a> type Item = (&'a str, EmbeddingWithNorm<'a>);
pub fn iter_with_norms(&self) -> IterWithNorms<'_>ⓘNotable traits for IterWithNorms<'a>impl<'a> Iterator for IterWithNorms<'a> type Item = (&'a str, EmbeddingWithNorm<'a>);
impl<'a> Iterator for IterWithNorms<'a> type Item = (&'a str, EmbeddingWithNorm<'a>);
Get an iterator over triples of words, embeddings, and norms.
Returns an iterator that returns triples of:
- A word.
- Its word embedding.
- The original norm of the embedding before normalization to a unit vector.
The original embedding can be reconstructed by multiplying all embedding components by the original norm.
If the model does not have associated norms, the norm is always 1.
impl<I, S> Embeddings<SubwordVocab<I>, S> where
I: BucketIndexer,
S: Storage + CloneFromMapping,
impl<I, S> Embeddings<SubwordVocab<I>, S> where
I: BucketIndexer,
S: Storage + CloneFromMapping,
Convert to explicitly indexed subword Embeddings.
Try to convert to explicitly indexed subword embeddings.
Conversion fails if the wrapped vocabulary is SimpleVocab
, FloretSubwordVocab
or
already an ExplicitSubwordVocab
.
Trait Implementations
Perform an analogy query. Read more
fn embedding_similarity_masked(
&self,
query: ArrayView1<'_, f32>,
limit: usize,
skip: &HashSet<&str>
) -> Option<Vec<WordSimilarityResult<'_>>>
fn embedding_similarity_masked(
&self,
query: ArrayView1<'_, f32>,
limit: usize,
skip: &HashSet<&str>
) -> Option<Vec<WordSimilarityResult<'_>>>
Find words that are similar to the query embedding while skipping certain words. Read more
fn embedding_similarity(
&self,
query: ArrayView1<'_, f32>,
limit: usize
) -> Option<Vec<WordSimilarityResult<'_>>>
fn embedding_similarity(
&self,
query: ArrayView1<'_, f32>,
limit: usize
) -> Option<Vec<WordSimilarityResult<'_>>>
Find words that are similar to the query embedding. Read more
Performs the conversion.
Performs the conversion.
Performs the conversion.
Performs the conversion.
Performs the conversion.
Performs the conversion.
impl From<Embeddings<SubwordVocab<ExplicitIndexer>, MmapArray>> for Embeddings<VocabWrap, StorageWrap>
impl From<Embeddings<SubwordVocab<ExplicitIndexer>, MmapArray>> for Embeddings<VocabWrap, StorageWrap>
Performs the conversion.
Performs the conversion.
Performs the conversion.
impl From<Embeddings<SubwordVocab<ExplicitIndexer>, NdArray>> for Embeddings<VocabWrap, StorageWrap>
impl From<Embeddings<SubwordVocab<ExplicitIndexer>, NdArray>> for Embeddings<VocabWrap, StorageWrap>
Performs the conversion.
impl From<Embeddings<SubwordVocab<ExplicitIndexer>, NdArray>> for Embeddings<VocabWrap, StorageViewWrap>
impl From<Embeddings<SubwordVocab<ExplicitIndexer>, NdArray>> for Embeddings<VocabWrap, StorageViewWrap>
Performs the conversion.
Performs the conversion.
impl From<Embeddings<SubwordVocab<FastTextIndexer>, MmapArray>> for Embeddings<VocabWrap, StorageWrap>
impl From<Embeddings<SubwordVocab<FastTextIndexer>, MmapArray>> for Embeddings<VocabWrap, StorageWrap>
Performs the conversion.
Performs the conversion.
Performs the conversion.
impl From<Embeddings<SubwordVocab<FastTextIndexer>, NdArray>> for Embeddings<VocabWrap, StorageWrap>
impl From<Embeddings<SubwordVocab<FastTextIndexer>, NdArray>> for Embeddings<VocabWrap, StorageWrap>
Performs the conversion.
impl From<Embeddings<SubwordVocab<FastTextIndexer>, NdArray>> for Embeddings<VocabWrap, StorageViewWrap>
impl From<Embeddings<SubwordVocab<FastTextIndexer>, NdArray>> for Embeddings<VocabWrap, StorageViewWrap>
Performs the conversion.
Performs the conversion.
impl From<Embeddings<SubwordVocab<FloretIndexer>, MmapArray>> for Embeddings<VocabWrap, StorageWrap>
impl From<Embeddings<SubwordVocab<FloretIndexer>, MmapArray>> for Embeddings<VocabWrap, StorageWrap>
Performs the conversion.
impl From<Embeddings<SubwordVocab<FloretIndexer>, MmapArray>> for Embeddings<VocabWrap, StorageViewWrap>
impl From<Embeddings<SubwordVocab<FloretIndexer>, MmapArray>> for Embeddings<VocabWrap, StorageViewWrap>
Performs the conversion.
Performs the conversion.
impl From<Embeddings<SubwordVocab<FloretIndexer>, NdArray>> for Embeddings<VocabWrap, StorageWrap>
impl From<Embeddings<SubwordVocab<FloretIndexer>, NdArray>> for Embeddings<VocabWrap, StorageWrap>
Performs the conversion.
impl From<Embeddings<SubwordVocab<FloretIndexer>, NdArray>> for Embeddings<VocabWrap, StorageViewWrap>
impl From<Embeddings<SubwordVocab<FloretIndexer>, NdArray>> for Embeddings<VocabWrap, StorageViewWrap>
Performs the conversion.
impl From<Embeddings<SubwordVocab<FloretIndexer>, QuantizedArray>> for Embeddings<VocabWrap, StorageWrap>
impl From<Embeddings<SubwordVocab<FloretIndexer>, QuantizedArray>> for Embeddings<VocabWrap, StorageWrap>
Performs the conversion.
impl From<Embeddings<SubwordVocab<HashIndexer<FnvHasher>>, MmapArray>> for Embeddings<VocabWrap, StorageWrap>
impl From<Embeddings<SubwordVocab<HashIndexer<FnvHasher>>, MmapArray>> for Embeddings<VocabWrap, StorageWrap>
Performs the conversion.
impl From<Embeddings<SubwordVocab<HashIndexer<FnvHasher>>, MmapArray>> for Embeddings<VocabWrap, StorageViewWrap>
impl From<Embeddings<SubwordVocab<HashIndexer<FnvHasher>>, MmapArray>> for Embeddings<VocabWrap, StorageViewWrap>
Performs the conversion.
Performs the conversion.
impl From<Embeddings<SubwordVocab<HashIndexer<FnvHasher>>, NdArray>> for Embeddings<VocabWrap, StorageWrap>
impl From<Embeddings<SubwordVocab<HashIndexer<FnvHasher>>, NdArray>> for Embeddings<VocabWrap, StorageWrap>
Performs the conversion.
impl From<Embeddings<SubwordVocab<HashIndexer<FnvHasher>>, NdArray>> for Embeddings<VocabWrap, StorageViewWrap>
impl From<Embeddings<SubwordVocab<HashIndexer<FnvHasher>>, NdArray>> for Embeddings<VocabWrap, StorageViewWrap>
Performs the conversion.
impl From<Embeddings<SubwordVocab<HashIndexer<FnvHasher>>, QuantizedArray>> for Embeddings<VocabWrap, StorageWrap>
impl From<Embeddings<SubwordVocab<HashIndexer<FnvHasher>>, QuantizedArray>> for Embeddings<VocabWrap, StorageWrap>
Performs the conversion.
Performs the conversion.
Performs the conversion.
fn quantize_using<T, R>(
&self,
n_subquantizers: usize,
n_subquantizer_bits: u32,
n_iterations: usize,
n_attempts: usize,
normalize: bool,
rng: R
) -> Result<Embeddings<V, QuantizedArray>> where
T: TrainPq<f32>,
R: CryptoRng + RngCore + SeedableRng + Send,
fn quantize_using<T, R>(
&self,
n_subquantizers: usize,
n_subquantizer_bits: u32,
n_iterations: usize,
n_attempts: usize,
normalize: bool,
rng: R
) -> Result<Embeddings<V, QuantizedArray>> where
T: TrainPq<f32>,
R: CryptoRng + RngCore + SeedableRng + Send,
Quantize the embedding matrix using the provided RNG. Read more
Read embeddings in the fastText format.
Read embeddings in the fastText format lossily. Read more
Read embeddings in the floret format.
Read the embeddings from the given buffered reader.
Read the embeddings from the given buffered reader. Read more
Read the embeddings from the given buffered reader.
Read the embeddings from the given buffered reader. Read more
Find words that are similar to the query word. Read more
impl<W, S> WriteFastText<W> for Embeddings<FastTextSubwordVocab, S> where
W: Write,
S: Storage,
impl<W, S> WriteFastText<W> for Embeddings<FastTextSubwordVocab, S> where
W: Write,
S: Storage,
Read embeddings in the floret format.
Auto Trait Implementations
impl<V, S> RefUnwindSafe for Embeddings<V, S> where
S: RefUnwindSafe,
V: RefUnwindSafe,
impl<V, S> Send for Embeddings<V, S> where
S: Send,
V: Send,
impl<V, S> Sync for Embeddings<V, S> where
S: Sync,
V: Sync,
impl<V, S> Unpin for Embeddings<V, S> where
S: Unpin,
V: Unpin,
impl<V, S> UnwindSafe for Embeddings<V, S> where
S: UnwindSafe,
V: UnwindSafe,
Blanket Implementations
Mutably borrows from an owned value. Read more