Struct UnicodeWordSplitter

pub struct UnicodeWordSplitter {}

Split text into words according to the Unicode definition of what a word is. While not perfect, it should work well enough as an easy starting point.

Uses UnicodeSegmentation::split_sentence_bounds under the hood.

Implementations

impl UnicodeWordSplitter

pub fn new() -> Self

Create a new UnicodeWordSplitter instance

Trait Implementations

impl Clone for UnicodeWordSplitter

fn clone(&self) -> UnicodeWordSplitter

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for UnicodeWordSplitter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for UnicodeWordSplitter

fn default() -> UnicodeWordSplitter

Returns the “default value” for a type.

impl Segmenter for UnicodeWordSplitter

type SubdivisionIter<'a> = IntoIter<SegmentedToken<'a>>

The iterator type returned by the subdivide function if it has multiple results.

fn subdivide<'a>( &self, token: SegmentedToken<'a>, ) -> UseOrSubdivide<SegmentedToken<'a>, IntoIter<SegmentedToken<'a>>>

A method that should split the given token into zero, one or more subtokens.