Trait finalfusion::compat::text::ReadTextDims [−][src]
pub trait ReadTextDims<R> where
Self: Sized,
R: BufRead, {
fn read_text_dims(reader: &mut R) -> Result<Self>;
fn read_text_dims_lossy(reader: &mut R) -> Result<Self>;
}Expand description
Method to construct Embeddings from a text file with dimensions.
This trait defines an extension to Embeddings to read the word embeddings
from a text stream. The text must contain as the first line the shape of
the embedding matrix:
vocab_size n_components
The remainder of the stream should contain one word embedding per line in the following format:
word0 component_1 component_2 … component_n
Required methods
fn read_text_dims(reader: &mut R) -> Result<Self>
fn read_text_dims(reader: &mut R) -> Result<Self>
Read the embeddings from the given buffered reader.
fn read_text_dims_lossy(reader: &mut R) -> Result<Self>
fn read_text_dims_lossy(reader: &mut R) -> Result<Self>
Read the embeddings from the given buffered reader.
In contrast to read_text_dims, this constructor does not
fail if a token contains invalid UTF-8. Instead, it will
replace invalid UTF-8 characters by the replacement
character.