Module tokenizer 

CSS tokenizer, implementing CSS Syntax Module Level 3 §3 (Tokenizing and Parsing CSS) and §4 (Tokenization).

This module only turns a &str into a stream of tokens. It has no knowledge of the Media Queries grammar (feature names, and/or/ not, range syntax, …) — that belongs to the parser (phase 03).

Structs

Tokenizer

A tokenizer over a preprocessed input stream, per spec §4.3.

Enums

HashType

Type flag of a Token::Hash (spec §4.3.1, the # branch).

NumericType

Numeric type flag, as used by Token::Number and Token::Dimension (spec §4.3.12, “Consume a number”).

Token

A single CSS token, per spec §4.3.1 (“Consume a token”) and the token types defined at the start of §4.3.

Functions

tokenize

Tokenizes input into a Vec<Token>, ending with a single Token::Eof. Convenience wrapper around Tokenizer for callers that don’t need streaming/lazy tokenization.