Enum Chunking 

#[non_exhaustive]
pub enum Chunking {
    Word,
    Line,
    Regex(Regex),
    UnicodeWords,
    Detector(ChunkDetector),
}

How buffered text is split into chunks.

Variants (Non-exhaustive)

Non-exhaustive enums could have additional variants added in future. Therefore, when matching against variants of non-exhaustive enums, an extra wildcard arm must be added to account for any future variants.

Word

A run of non-whitespace followed by whitespace (\S+\s+).

Line

Up to and including a run of newlines (\n+).

Regex(Regex)

The first match of a regular expression; the chunk spans from the buffer start to the end of the match. Empty matches fail the stream.

UnicodeWords

The first Unicode word-boundary segment (UAX #29), emitted immediately. Whitespace and punctuation are separate segments; locale dictionaries and ICU tailoring require an application-supplied detector.

Detector(ChunkDetector)

A custom detector. Lengths of zero, beyond the buffer or inside a character fail the smoothing stream.

Implementations

impl Chunking

pub fn detector( f: impl Fn(&str) -> Option<usize> + Send + Sync + 'static, ) -> Self

Custom detector.

pub fn detect(&self, buffer: &str) -> Option<usize>

Byte length of the next chunk in buffer, if complete.

Trait Implementations

impl Clone for Chunking

fn clone(&self) -> Chunking

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Chunking

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Chunking

fn default() -> Chunking

Returns the “default value” for a type.