pub fn tokenize(input: &str) -> Vec<Token>Expand description
Tokenize a CSS input string into a vector of tokens.
Implements the tokenization stage of CSS Syntax §3.1: construct a
tokenizer over input (after §5.3 input preprocessing, which the
tokenizer applies internally), then drain all tokens up to and
including <EOF-token>.
Returns the token stream without the trailing <EOF-token>. Parse
errors are currently discarded; a future API will expose them.
§Examples
use muskitty_css_parser::tokenize;
use muskitty_css_parser::Token;
let tokens = tokenize("color: red;");
assert!(matches!(tokens[0], Token::Ident(_)));
assert!(matches!(tokens[1], Token::Colon));
assert!(matches!(tokens[2], Token::Whitespace));
assert!(matches!(tokens[3], Token::Ident(_)));
assert!(matches!(tokens[4], Token::Semicolon));