Struct NaiveWordSplitter

pub struct NaiveWordSplitter {}

Naive word splitting that is based on tokens being cut where alphanimerics, space and symbols change from one type to another.

⚠️ This segmenter is unstable and may change significantly outside of semver gurantees

This currently isn’t all that useful and as the name says naive.

Please prefer the UnicodeWordSplitter.

Implementations

impl NaiveWordSplitter

pub fn new() -> Self

Create a new UnicodeWordSplitter instance

Trait Implementations

impl Clone for NaiveWordSplitter

fn clone(&self) -> NaiveWordSplitter

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for NaiveWordSplitter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for NaiveWordSplitter

fn default() -> NaiveWordSplitter

Returns the “default value” for a type.

impl Segmenter for NaiveWordSplitter

type SubdivisionIter<'a> = IntoIter<SegmentedToken<'a>>

The iterator type returned by the subdivide function if it has multiple results.

fn subdivide<'a>( &self, token: SegmentedToken<'a>, ) -> UseOrSubdivide<SegmentedToken<'a>, IntoIter<SegmentedToken<'a>>>

A method that should split the given token into zero, one or more subtokens.