Struct sentence::SentenceTokenizer

pub struct SentenceTokenizer {}

Sentence Tokenizer.

For now this is stateless. This library is in a very early state, but I intend to add preferences and dictionary lookup callback support. ("Intend to come back" are famous last words.)

Implementations

impl SentenceTokenizer

pub fn new() -> Self

Constructor.

pub fn tokenize(&self, sequence: &str) -> Vec<Token>

Turn a text sequence into a series of tokens.

use sentence::{SentenceTokenizer, Token, Punctuation};
let tokenizer = SentenceTokenizer::new();
let tokens = tokenizer.tokenize("Hello, world!");
assert_eq!(tokens, vec![
  Token::Word("Hello".into()),
  Token::Punctuation(Punctuation::Comma),
  Token::Word("world".into()),
  Token::Punctuation(Punctuation::Exclamation),
]);