[−][src]Crate sentence
Sentence, a library for lexing English language sentences into tokens for TTS purposes.
Usage:
use sentence::{SentenceTokenizer, Token, Punctuation}; let tokenizer = SentenceTokenizer::new(); let tokens = tokenizer.tokenize("Hello, world!"); assert_eq!(tokens, vec![ Token::Word("Hello".into()), Token::Punctuation(Punctuation::Comma), Token::Word("world".into()), Token::Punctuation(Punctuation::Exclamation), ]);
Structs
SentenceTokenizer | Sentence Tokenizer. |
Enums
Punctuation | Punctuation marks |
Token | A parsed token |