Enum imdb_index::NgramType

pub enum NgramType {
    Window,
    Edge,
}

The style of ngram extraction to use.

The same style of ngram extraction is always used at index time and at query time.

Each ngram type uses the ngram size configuration differently.

All ngram styles used Unicode codepoints as the definition of a character. For example, a 3-gram might contain up to 4 bytes, if it contains 3 Unicode codepoints that each require 4 UTF-8 code units.

Variants

Window

A windowing ngram.

This is the tradition style of ngram, where sliding window of size N is moved across the entire content to be index. For example, the 3-grams for the string homer are hom, ome and mer.

Edge

An edge ngram.

This style of ngram produces ever longer ngrams, where each ngram is anchored to the start of a word. Words are determined simply by splitting whitespace.

For example, the edge ngrams of homer simpson, where the max ngram size is 5, would be: hom, home, homer, sim, simp, simps. Generally, for this ngram type, one wants to use a large maximum ngram size. Perhaps somewhere close to the maximum number of ngrams in any word in the corpus.

Note that there is no way to set the minimum ngram size (which is 3).

Methods

impl NgramType

pub fn possible_names() -> &'static [&'static str]

Return all possible ngram types.

pub fn as_str(&self) -> &'static str

Return a string representation of this type.

Trait Implementations

impl PartialEq<NgramType> for NgramType

fn eq(&self, other: &NgramType) -> bool

This method tests for self and other values to be equal, and is used by ==.

#[must_use] fn ne(&self, other: &Rhs) -> bool

This method tests for !=.

impl Copy for NgramType

impl Eq for NgramType

impl Default for NgramType

fn default() -> NgramType

Returns the "default value" for a type.

impl Clone for NgramType

fn clone(&self) -> NgramType

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Hash for NgramType

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given [Hasher].

fn hash_slice<H>(data: &[Self], state: &mut H) where
    H: Hasher, 

Feeds a slice of this type into the given [Hasher].

impl Debug for NgramType

fn fmt(&self, f: &mut Formatter) -> Result

Formats the value using the given formatter.

impl Display for NgramType

fn fmt(&self, f: &mut Formatter) -> Result

Formats the value using the given formatter.

impl FromStr for NgramType

type Err = Error

The associated error which can be returned from parsing.

fn from_str(s: &str) -> Result<NgramType>

Parses a string s to return a value of this type.

impl<'de> Deserialize<'de> for NgramType

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
    __D: Deserializer<'de>, 

Deserialize this value from the given Serde deserializer.

impl Serialize for NgramType

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error> where
    __S: Serializer, 

Serialize this value into the given Serde serializer.