Struct Lexer 

pub struct Lexer;

An English-aware tokenizer.

Construct one with Lexer::new and call Lexer::lex. The struct holds no state, so a single instance is reusable and cheap to create.

use pos_tagger::Lexer;
let lexer = Lexer::new();
assert_eq!(lexer.lex("I made $5.42 today."), vec!["I", "made", "$", "5.42", "today", "."]);

Implementations

impl Lexer

pub fn new() -> Self

Create a lexer.

pub fn lex(&self, string: &str) -> Vec<String>

Split string into tokens.

Returns tokens in textual order. URLs, emails, long IDs, times, and numbers stay whole. Single punctuation marks split off as their own tokens. Whitespace-only fragments are dropped and every token is trimmed.

Trait Implementations

impl Clone for Lexer

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for Lexer

impl Debug for Lexer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Lexer

fn default() -> Self

Returns the “default value” for a type.