Expand description
MusKitty CSS Tokenizer
Implements the tokenization stage of the CSS Syntax Module Level 3 (§4.3 “Tokenizer Algorithms”). Extracted from muskitty-css as a standalone crate for independent versioning and publication.
The tokenizer consumes a stream of Unicode code points (after §5.3
preprocessing) and emits Tokens. These tokens are consumed by
downstream parsers (e.g. muskitty-css’s parser, muskitty-selectors)
to build CSS objects or selector ASTs.
§References
- CSS Syntax Module Level 3: https://drafts.csswg.org/css-syntax-3/
- Spec source (Markdown):
D:\CSSWG\css-syntax-3\Overview.md
Structs§
- CssTokenizer
- The CSS tokenizer (§4.3).
- Numeric
- A numeric value carried by
<number-token>,<percentage-token>, and<dimension-token>(§4.3.13).
Enums§
- Hash
Type - The type flag on a
<hash-token>(§4.1, §4.3.4). - State
- The current state of the CSS tokenizer.
- Token
- A CSS token (§4.1).
Traits§
- Tokenizer
- The CSS tokenizer.