1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
//! The alphabet the randomised property tests draw their sources from.
//!
//! Two suites generate sources this way — the checkpoint and incremental
//! properties in `src/incremental.rs`, and the whole-file properties in
//! `tests/properties.rs` — and they are meant to draw from the same alphabet: a
//! fragment worth generating against the whole-file scanner is worth generating
//! against the incremental one, because the incremental engine's promise is
//! that a restart reproduces what the whole-file scan would have said. One is a
//! unit test inside the crate and the other an integration test outside it, and
//! the only thing both can name is the crate's public surface, so the alphabet
//! lives here rather than being written out twice.
//!
//! It is `#[doc(hidden)]` and carries no stability promise: it is test support
//! that happens to have to be reachable from outside.
/// Single bytes that reach every built-in scanner's string, comment, here
/// document and template states rather than only the C-family delimiters.
///
/// A generator draws one of these against a much smaller weight of uniform
/// random bytes, so a delimiter arrives often enough for two of them to meet.
/// Each byte appears once and is drawn as often as the next; a caller that
/// wants one of them oftener says so with a weight of its own.
pub const BYTES: & = b"\n\r/*'\"#`{}<>=[]-|?\\$%()@:!~";
/// Multi-byte tokens a single-byte alphabet can never synthesise.
///
/// The preamble and directive rules only fire on whole words, so without these
/// the generated sources never reach the code paths that make a scan depend on
/// where in the document it starts.
///
/// The two triple-quote runs are here for the opposite reason: they are three
/// of one byte, which a per-byte alphabet reaches only by coincidence, and they
/// open a string that swallows newlines in Python, Kotlin, Java, and TOML —
/// which is exactly the state a restart must not be allowed to land inside.
/// Lua's long brackets are the same state behind four bytes rather than three,
/// and the levelled forms are here because a closing bracket of the wrong level
/// is content: without them a generated source that opens one practically never
/// closes it. The eight YAML fragments after them are block scalar headers and
/// the indented line that follows one — a body is the state a YAML restart must
/// never land inside, and the bytes that open one have to arrive in that order.
/// Three of the eight put the owner of that body on an earlier line than its
/// header: a line ending in `:` or in a bare `-`, and the node properties that
/// may stand between the two. That owner is the one thing a YAML line does not
/// say about itself, so it is the one thing a restart at a line start has to be
/// refused over. The `|+` header is the keep-chomped body whose trailing blank
/// lines are content. The five PHP fragments after those are its two tags, an
/// attribute, and a here document header with the line that closes one: PHP
/// mode is the state a restart must never land inside, and only a whole `<?php`
/// opens it. The Ruby fragments at the end are its two column-zero markers with
/// the line each of them needs, a here document header with the terminator that
/// ends one, a percent literal opener, and the interpolation boundary a here
/// document header may be written across: four of Ruby's tokens are spelled
/// with a byte that is also an operator, so only the whole opener reaches the
/// state, and an embedded document, a here document body and the DATA section
/// are three more states a restart must never land inside. The body a header
/// inside `"#{ ... }"` asks for belongs to the line the header stands on rather
/// than to the interpolation, and only a pool that can assemble that opener
/// generates the case at all. The two Zig fragments at the end are the whole
/// opener of a multiline string literal line — two of one byte, which a
/// per-byte alphabet reaches only by coincidence, and the only thing in the
/// language that hides a `//` without a quote — and the fourth slash that turns
/// a documentation comment back into an ordinary one. The five R fragments
/// after them are its raw string openers and closers and its roxygen marker:
/// the `r` that opens a raw string is a letter no per-byte alphabet carries, so
/// without them a generated R source never reaches the one literal in the
/// language whose delimiter is neither a quote nor a bracket alone, and the
/// dashed pair is there for the reason Lua's levelled brackets are — a closing
/// run of the wrong length is content. The three Dart fragments after those are
/// its raw string openers, whose `r` is a letter no per-byte alphabet carries
/// and which take neither an escape nor an interpolation, and the interpolation
/// opener itself: a `${` turns the inside of a string back into code, and a
/// comment written there is one more state a restart must not land inside. The
/// five Swift fragments after them are its raw-string and extended-regex
/// delimiters with the escape the hashes rename: `#"` and `"#` open and close
/// the raw string whose escape is `\#(`, and `#/` and `/#` the regular
/// expression literal that may hold an unescaped `/` and may span lines. Each
/// is two or more bytes that have to arrive in that order, and each opens a
/// state a restart must not land inside. The seven C# fragments at the end are
/// its four string openers, the brace pair that is an escape in one form and a
/// hole in another, and two directive words: an `@` and a quote and a `$@` and
/// a quote carry line breaks, a `$` and a quote carries a hole that may, a run
/// of two `$` in front of three quotes needs two braces to open one, and a line
/// whose first non-blank byte opens a directive is lexed by rules of its own.
/// Each is a state a restart must not land inside, and none of them is a shape
/// a per-byte alphabet assembles.
pub const TOKENS: & = &;