Skip to main content

tokenize

Function tokenize 

Source
pub fn tokenize(input: &str) -> Vec<Token>
Expand description

Tokenize a CSS input string into a vector of tokens.

Implements the tokenization stage of CSS Syntax §3.1: construct a tokenizer over input (after §5.3 input preprocessing, which the tokenizer applies internally), then drain all tokens up to and including <EOF-token>.

Returns the token stream without the trailing <EOF-token>. Parse errors are currently discarded; a future API will expose them.

§Examples

use muskitty_css_parser::tokenize;
use muskitty_css_parser::Token;

let tokens = tokenize("color: red;");
assert!(matches!(tokens[0], Token::Ident(_)));
assert!(matches!(tokens[1], Token::Colon));
assert!(matches!(tokens[2], Token::Whitespace));
assert!(matches!(tokens[3], Token::Ident(_)));
assert!(matches!(tokens[4], Token::Semicolon));