Expand description
MusKitty HTML5 Tokenizer
Implements the tokenization stage of the WHATWG HTML parsing model (§13.2.5 Tokenization).
The tokenizer consumes a stream of Unicode code points and emits tokens (start tags, end tags, comments, characters, DOCTYPEs, EOF). These tokens are consumed by the tree construction stage to build the DOM tree.
§References
- WHATWG HTML Living Standard: https://html.spec.whatwg.org/multipage/parsing.html
- WPT test suite: https://github.com/web-platform-tests/wpt/tree/master/html/syntax/parsing
Structs§
- Doctype
Token - A DOCTYPE token.
- Html
Tokenizer - A concrete HTML tokenizer.
- TagToken
- A tag token (used for both start and end tags, distinguished by
TagKind).
Enums§
- Parse
Error - Parse errors defined by WHATWG §13.2.5.
- State
- Tokenizer states.
- TagKind
- Distinguishes start tags from end tags.
- Token
- A token emitted by the tokenizer.
Traits§
- Tokenizer
- The HTML tokenizer.