Struct finalfusion::subword::ExplicitIndexer

pub struct ExplicitIndexer { /* fields omitted */ }

Indexer for explicitly stored NGrams.

Methods

impl ExplicitIndexer

pub fn ngrams(&self) -> &[String]

impl ExplicitIndexer

pub fn new(ngrams: impl Into<Vec<String>>) -> Self

Construct a new explicit indexer.

Panics when there are duplicate ngrams.

pub fn new_with_indices(ngram_tuples: Vec<(String, u64)>) -> Self

Construct a new explicit indexer with given indices.

The (String, u64) tuples resemble the original subword -> index mapping. This mapping does not need to be perfect, i.e. multiple subwords can map to the same index as it is common with bucketed indexing.

This constructor numbers the original indices as they appear and assigns a new index accordingly. After construction, subwords that originally had the same index will still be indexed by a common number. It is guaranteed that the new indices cover (0..n_original_indices) where n_original_indices is the number of unique indices in the subword -> index mapping.

Panics when there are duplicate ngrams.

Trait Implementations

impl Clone for ExplicitIndexer

fn clone(&self) -> ExplicitIndexer

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ExplicitIndexer

fn fmt(&self, f: &mut Formatter) -> Result

Formats the value using the given formatter.

impl Eq for ExplicitIndexer

impl Indexer for ExplicitIndexer

fn index_ngram(&self, ngram: &StrWithCharLen) -> Option<u64>

Map an n-gram to an index in the subword embedding matrix.

fn upper_bound(&self) -> u64

Return the (exclusive) upper bound of this indexer.

fn infallible() -> bool

Indicates whether this Indexer never fails to produce an index.

impl PartialEq<ExplicitIndexer> for ExplicitIndexer

fn eq(&self, other: &ExplicitIndexer) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &ExplicitIndexer) -> bool

This method tests for !=.

impl StructuralEq for ExplicitIndexer

impl StructuralPartialEq for ExplicitIndexer