Struct SentenceTokenizer 

pub struct SentenceTokenizer { /* private fields */ }

Sentence tokenizer that splits text into sentences.

Uses punctuation-based rules to detect sentence boundaries, handling common abbreviations and edge cases.

Examples

use aprender::text::tokenize::SentenceTokenizer;

let tokenizer = SentenceTokenizer::new();

let sentences = tokenizer.split("Hello world. How are you? I'm fine!");
assert_eq!(sentences, vec!["Hello world.", "How are you?", "I'm fine!"]);

Implementations

impl SentenceTokenizer

pub fn new() -> Self

Create a new sentence tokenizer with default abbreviations.

pub fn split(&self, text: &str) -> Vec<String>

Split text into sentences.

Trait Implementations

impl Clone for SentenceTokenizer

fn clone(&self) -> SentenceTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for SentenceTokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for SentenceTokenizer

fn default() -> SentenceTokenizer

Returns the “default value” for a type.