Function tokenize 

pub fn tokenize(input: &str) -> Vec<Token>

Lex a structprop input string into a flat Vec of Tokens.

Comments (# … \n) and insignificant whitespace (spaces, tabs, carriage returns, and newlines) are discarded. The returned vector always ends with Token::Eof.

Examples

use serde_structprop::lexer::{tokenize, Token};

let tokens = tokenize("key = value");
assert_eq!(tokens, vec![
    Token::Term("key".into()),
    Token::Eq,
    Token::Term("value".into()),
    Token::Eof,
]);