pub enum Token {
Show 27 variants
Ident(String),
Function(String),
AtKeyword(String),
Hash(String, HashType),
String(String),
BadString,
Url(String),
BadUrl,
Delim(char),
Number(Numeric),
Percentage(Numeric),
Dimension(Numeric, String),
UnicodeRange(Option<u32>, Option<u32>),
Whitespace,
Comment(String),
Colon,
Semicolon,
Comma,
OpenBracket,
CloseBracket,
OpenParen,
CloseParen,
OpenBrace,
CloseBrace,
Cdo,
Cdc,
Eof,
}Expand description
A CSS token (§4.1).
Variants correspond to the token types defined by the CSS Syntax Module §4.1 railroad diagrams. Each variant’s doc comment cites the relevant normative section.
Variants§
Ident(String)
<ident-token> (§4.1). An identifier such as color, auto,
-webkit-flex. The value is the decoded ident sequence (escapes
resolved, §4.3.12).
Function(String)
<function-token> (§4.1). An identifier immediately followed by
(, e.g. translate(, var(. The value is the function name
(without the parenthesis).
AtKeyword(String)
<at-keyword-token> (§4.1). @ followed by an ident sequence,
e.g. @media, @import. The value is the keyword (without @).
Hash(String, HashType)
<hash-token> (§4.1). # followed by an ident sequence or hex
digits, e.g. #main, #ff00aa. The HashType flag indicates
whether the hash is an “id” hash (would-start-an-ident-sequence),
an “unrestricted” hash (hex digits that don’t form an ident), or
“unknown” for hashes that don’t match either pattern.
String(String)
<string-token> (§4.1). A quoted string ("..." or '...') with
escapes resolved. The value is the string’s contents without the
surrounding quotes.
BadString
<bad-string-token> (§4.1). Emitted when a string contains an
unescaped newline (§4.3.5).
Url(String)
<url-token> (§4.1). The url(...) form with an unquoted or
quoted URL. The value is the decoded URL contents.
BadUrl
<bad-url-token> (§4.1). Emitted when a URL contains an invalid
escape or unterminated quoted segment (§4.3.6).
Delim(char)
<delim-token> (§4.1). A single code point not consumed by any
other token type, e.g. >, +, ~ (when not part of ~=),
! (when not part of !=).
Number(Numeric)
<number-token> (§4.1). A numeric value, possibly signed, possibly
with a fractional part or scientific notation. The Numeric
carries the parsed value and integer/number flag.
Percentage(Numeric)
<percentage-token> (§4.1). A number immediately followed by %.
Dimension(Numeric, String)
<dimension-token> (§4.1). A number immediately followed by a unit
(an ident sequence), e.g. 10px, 1.5em, -30deg. The value
carries the Numeric and the unit string.
UnicodeRange(Option<u32>, Option<u32>)
<unicode-range-token> (§4.1). A range of Unicode code points in
the U+ form, e.g. U+1234, U+12-34FF, U+12??. The two
Option<u32> values are the start and end of the range. Either
may be None only in malformed inputs that the spec still accepts
as a unicode-range; in practice both are Some for well-formed
ranges.
Whitespace
<whitespace-token> (§4.1). A run of one or more whitespace code
points (space, tab, newline, §4.2 definition of whitespace).
Comment(String)
<comment-token> (§4.1). The contents of a /* ... */ comment
(without the delimiters). Comment tokens are only emitted in
“tokenizer-friendly” mode; the default stylesheet tokenizer
discards them. This implementation emits them so callers can
choose.
Colon
<colon-token> (§4.1). :.
Semicolon
<semicolon-token> (§4.1). ;.
Comma
<comma-token> (§4.1). ,.
OpenBracket
<[-token> (§4.1). [.
CloseBracket
<]-token> (§4.1). ].
OpenParen
<(-token> (§4.1). (.
CloseParen
<)-token> (§4.1). ).
OpenBrace
<{-token> (§4.1). {.
CloseBrace
<}-token> (§4.1). }.
Cdo
<CDO-token> (§4.1). The literal <!-- (Comment Declaration
Open). Kept for backwards compatibility with CSS 1/2.1’s practice
of wrapping stylesheets in <!-- … --> to hide them from
ancient browsers. At the tokenizer level this is a single token
spanning all four code points.
Cdc
<CDC-token> (§4.1). The literal --> (Comment Declaration
Close). See Token::Cdo.
Eof
<EOF-token> (§5.3). Emitted once when the input stream is
exhausted. [Tokenizer::next_token] returns None after this.