Struct Splitter

pub struct Splitter { /* private fields */ }

A struct for splitting texts into segments based on the most desirable separator found.

Examples

use semchunk_rs::Splitter;
let splitter = Splitter::default();
let text = "Hello World\nGoodbye World";
let (separator, is_whitespace, segments) = splitter.split_text(text);
assert_eq!(separator, "\n");
assert!(is_whitespace);
assert_eq!(segments, vec!["Hello World", "Goodbye World"]);

Implementations

impl Splitter

pub fn split_text<'a>(&self, text: &'a str) -> (&'a str, bool, Vec<&'a str>)

Splits the given text into segments based on the most desirable separator found.

The method prioritizes separators in the following order:

1. The largest sequence of newlines and/or carriage returns.
2. The largest sequence of tabs.
3. The largest sequence of whitespace characters.
4. A semantically meaningful non-whitespace separator.

If no semantically meaningful separator is found, the text is split into individual characters.

Arguments

• text - A string slice that holds the text to be split.

Returns

A tuple containing:

• The separator used for splitting the text.
• A boolean indicating whether the separator is whitespace.
• A vector of string slices representing the segments of the split text.

Examples

use semchunk_rs::Splitter;
let splitter = Splitter::default();
let text = "Hello World\nGoodbye World";
let (separator, is_whitespace, segments) = splitter.split_text(text);
assert_eq!(separator, "\n");
assert!(is_whitespace);
assert_eq!(segments, vec!["Hello World", "Goodbye World"]);

Trait Implementations

impl Debug for Splitter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Splitter

fn default() -> Self

Returns the “default value” for a type.