Struct AugmentationClassify

pub struct AugmentationClassify {}

An augmenter that rewrites the SegmentedToken::kind field to match reality.

It does so by reading the token text (preferring the normalized text) and applying heuristics based on the unicode GeneralCategoryGroup of the characters it contains.

The following heuristics are applied in the given order:

1. If it contains Letters or Numbers -> SegmentedTokenKind::AlphaNumeric
2. If it contains Symbols or Other -> SegmentedTokenKind::Symbol
3. If it contains Punctuation or Separators -> SegmentedTokenKind::Separator

Exceptions from usual unicode classification: \n and \0 are seperators.

The Mark category is ignored. If none of the heuristics apply the token kind is reset to None.

Implementations

impl AugmentationClassify

pub fn new() -> Self

Create a new classify augmenter with default settings.

Trait Implementations

impl Augmenter for AugmentationClassify

fn augment<'a>(&self, token: SegmentedToken<'a>) -> SegmentedToken<'a>

Apply augmentation function to the given token and return it.

impl Clone for AugmentationClassify

fn clone(&self) -> AugmentationClassify

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for AugmentationClassify

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for AugmentationClassify

fn default() -> AugmentationClassify

Returns the “default value” for a type.