Struct nlprule::tokenizer::TokenizerOptions

pub struct TokenizerOptions {
    pub allow_errors: bool,
    pub retain_last: bool,
    pub use_compound_split_heuristic: bool,
    pub always_add_lower_tags: bool,
    pub ids: Vec<String>,
    pub ignore_ids: Vec<String>,
    pub known_failures: Vec<String>,
    pub extra_tags: Vec<String>,
    pub extra_split_chars: Vec<char>,
    pub extra_join_regexes: Vec<SerializeRegex>,
}

Options for a tokenizer.

Fields

allow_errors: bool

Whether to allow errors while constructing the tokenizer.

retain_last: bool

Whether to retain the last tag if disambiguation leads to an empty tag. Language-specific in LT so it has to be an option.

use_compound_split_heuristic: bool

Whether to use a heuristic to split potential compound words.

always_add_lower_tags: bool

Whether to always add tags for a lowercase version of the word when assigning part-of-speech tags.

ids: Vec<String>

Disambiguation Rule IDs to use in this tokenizer.

ignore_ids: Vec<String>

Disambiguation Rule IDs to ignore in this tokenizer.

known_failures: Vec<String>

Specific examples in the notation {id}:{example_index} which are known to fail.

extra_tags: Vec<String>

Used part-of-speech tags which are not in the tagger dictionary.

extra_split_chars: Vec<char>

Extra language-specific characters to split text on.

extra_join_regexes: Vec<SerializeRegex>

Extra language-specific Regexes of which the matches will not be split into multiple tokens.

Trait Implementations

impl Default for TokenizerOptions

fn default() -> Self

Returns the "default value" for a type.

impl<'de> Deserialize<'de> for TokenizerOptions

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
    __D: Deserializer<'de>, 

Deserialize this value from the given Serde deserializer.

impl Serialize for TokenizerOptions

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error> where
    __S: Serializer, 

Serialize this value into the given Serde serializer.