pub fn tokenize(input: &str) -> Vec<Token>Expand description
Tokenizes the input string into a vector of tokens.
§Arguments
input- The input string to tokenize.
§Returns
A vector of tokens parsed from the input.
§Examples
use crate::lexer::{Token, tokenize}; let tokens = tokenize(“x <- 10 + y”); assert_eq!( tokens, vec![ Token::Identifier(“x”.to_string()), Token::Assign, Token::Number(10.0), Token::Plus, Token::Identifier(“y”.to_string()), Token::EOF, ] );