1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74

//! A library for reading, writing, and using word embeddings.
//!
//! finalfusion allows you to read, write, and use
//! word2vec/[GloVe](https://nlp.stanford.edu/projects/glove/)
//! embeddings and read [fastText](https://fasttext.cc/) embeddings.
//! finalfusion uses *finalfusion* as its native data format, which
//! has several benefits over the word2vec, GloVe, and fastText
//! formats.
//!
//! ## Reading finalfusion embeddings
//!
//! finalfusion embeddings can be read with the `read_embeddings`
//! method, which expects a reader that implements the `BufRead`
//! trait.
//!
//! Since finalfusion supports various types of vocabularies and
//! embedding matrix (storage) formats, these should be specified
//! as type parameters of the `Embeddings` type. However, typically
//! one would want to read finalfusion embeddings with any type of
//! vocabulary or embedding matrix. For this purpose, the `VocabWrap`
//! and `StorageWrap` types are provided, which wrap any type of
//! vocabulary and embeddung matrix.
//!
//! We can thus load a finalfusion format and retrieve an embedding
//! as follows:
//!
//! ```
//! use std::fs::File;
//! use std::io::BufReader;
//!
//! use finalfusion::prelude::*;
//!
//! let mut reader = BufReader::new(File::open("testdata/similarity.fifu").unwrap());
//!
//! // Read the embeddings.
//! let embeddings: Embeddings<VocabWrap, StorageWrap> =
//!     Embeddings::read_embeddings(&mut reader)
//!     .unwrap();
//!
//! // Look up an embedding.
//! let embedding = embeddings.embedding("Berlin");
//! ```
//!
//! For performing analogy/similarity queries on the embedding
//! matrix, we need an embedding matrix which can act as a view.
//! In that case one should use `StorageViewWrap` in place of
//! `StorageWrap`. `StorageViewWrap` is only supported for a
//! subset of embedding matrix types -- in particular, quantized
//! matrices cannot be used as a view.
//!
//! ## Reading other embedding formats
//!
//! Consult the documentation of the `fasttext`, `text` and
//! `word2vec` modules for information on how to read fastText,
//! GloVe, and word2vec embeddings.

mod chunks;
pub use chunks::{metadata, norms, storage, vocab};

pub mod compat;

pub mod embeddings;

pub mod error;

pub mod io;

pub mod prelude;

pub mod similarity;

pub mod subword;

pub(crate) mod util;