1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
//! A token as the analysis layer sees it: owned text, file coordinates, and nothing else.
//!
//! The syntax tree stores tokens as ranges into the file, which is what makes it lossless and cheap.
//! That is the wrong shape for the analysis layer to work in, for two reasons:
//!
//! * **Expansion produces tokens that are in no file.** A `#define` body pasted into a call site, or a
//! `##` joining two spellings, has no range to point at. Anything holding a `SourceRange` would have
//! to invent one.
//! * **The tree is borrowed; analysis results are not.** A resolved macro table outlives the parse it
//! came from — that is the point of caching it — so it cannot borrow a `&str` that the tree owns.
//!
//! So tokens are flattened once, into owned `Box<str>` text. A directive is a few dozen tokens at
//! most, and there are as many directives as there are lines beginning with `#`, so the copying is
//! bounded by the number of directives rather than by the size of the file.
use ;
/// One token, with its text detached from the tree.
///
/// Not `Hash`: [`SourceRange`] is not, and a token is not a key in anything — the macro table is keyed
/// by name, and directives by position.
/// Every token of a node, in order, trivia included.
///
/// Trivia is kept rather than filtered because directive parsing needs it: whether `MAX` in
/// `#define MAX(x)` is function-like depends on whether a `(` follows it *immediately*, and "immediately"
/// is a question about the gap between two tokens. A caller that wants the significant tokens filters
/// them afterwards, with [`is_trivia`](cpp_parser::is_trivia).
/// Is this token invisible to the grammar (whitespace, newline, splice, comment)?
/// Does this token end a logical line?
///
/// A `LineContinuation` does not: `#define FOO \<newline> bar` is one directive, which is exactly why
/// the lexer gives the splice a kind of its own instead of folding it into whitespace.
/// Is this the `#` that begins a directive, as opposed to a `#` inside a macro body?
/// Is this token a keyword — a word C++ reserves, but which is an ordinary word inside a directive?
///
/// `#define private public` and `#define true 1` are used in real test suites, and neither means
/// anything special in a directive. The lexer has no way to know it is inside one, so this is the test
/// that recovers the distinction.
/// The `CppKind` of a token kind, for callers that need to compare against tree kinds.