Struct tokenizer_py::Tokenizer

pub struct Tokenizer { /* private fields */ }

A struct that can tokenize a string into tokens.

Examples

use tokenizer_py::{Tokenizer, Token};

let tokenizer = Tokenizer::new("hello world".to_string());
let tokens = tokenizer.tokenize().unwrap();
assert_eq!(tokens, vec![
    Token::Name("hello".to_string()),
    Token::Name("world".to_string()),
    Token::EndMarker,
]);

Implementations

impl Tokenizer

pub const fn new(text: String) -> Tokenizer

Creates a new tokenizer that will tokenize the given text.

Examples

use tokenizer_py::Tokenizer;

let tokenizer = Tokenizer::new("hello world".to_string());

pub fn tokenize(&self) -> Result<Vec<Token>, TokenizerError>

Tokenizes the text that was provided to the tokenizer’s constructor.

Examples

use tokenizer_py::{Token, Tokenizer};

let tokenizer = Tokenizer::new("hello\nworld".to_string());
let tokens = tokenizer.tokenize().unwrap();

assert_eq!(tokens, vec![
    Token::Name("hello".to_string()),
    Token::NewLine,
    Token::Name("world".to_string()),
    Token::EndMarker,
]);

Trait Implementations

impl Debug for Tokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for Tokenizer

fn eq(&self, other: &Tokenizer) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for Tokenizer

impl StructuralPartialEq for Tokenizer