1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
//! Tokenizer trait definition.
//!
//! CSS Syntax Module Level 3 §4.3 "Tokenizer Algorithms".
use crate;
/// The CSS tokenizer.
///
/// Implements the tokenization algorithm of CSS Syntax §4.3. The tokenizer
/// consumes Unicode code points from an input stream (preprocessed per
/// §5.3 to normalize newlines) and emits [`Token`]s.
///
/// # Algorithm structure
///
/// Unlike the HTML tokenizer's explicit state machine, the CSS tokenizer
/// is recursive-descent: a single entry point `consume_a_token` (§4.3.1)
/// dispatches to sub-algorithms (`consume_an_ident_like_token` §4.3.4,
/// `consume_a_numeric_token` §4.3.3, `consume_a_string_token` §4.3.5,
/// `consume_a_url_token` §4.3.6) which in turn call primitives
/// (`consume_an_escaped_code_point` §4.3.7, `consume_an_ident_sequence`
/// §4.3.12, `consume_a_number` §4.3.13).
///
/// # Reentrancy
///
/// The CSS tokenizer is *not* reentrant in the same way the HTML
/// tokenizer is: CSS has no analogue of HTML's content-model switching
/// (RCDATA / RAWTEXT / ScriptData). The state is therefore minimal —
/// just "consuming" vs. "EOF emitted". Future CSSOM incremental-parsing
/// use cases may require reentrancy; the trait exposes
/// [`state`](Tokenizer::state) and [`set_state`](Tokenizer::set_state)
/// to support that future need.