Skip to main content

Module tokens

Module tokens 

Source
Expand description

The token stream a formatter needs: every token, and the trivia between them.

The parser’s job is to build a tree, so it skips trivia; a formatter’s job is to rewrite trivia, so it needs the trivia to be data. cpp_parser::lex gives the full stream (Whitespace, Newline, LineContinuation, LineComment, BlockComment are all there), and this module splits it into

tokens[i]   the i-th significant token (its text is a slice of the source)
gaps[i]     the trivia between tokens[i-1] and tokens[i]: the newlines, and **the comments in order**
trailing    the trivia after the last token (usually the file's final newline)

§Why the comments are separated from the whitespace

Because a comment is content: the formatter may move it to a new line or indent it, but it must not change one byte of its text. Keeping comments as ordered items inside the gap lets the layout put each one on its own line (or after the previous token, when it was written that way) without ever looking inside the comment’s text.

Structs§

Comment
One comment, and where it stood relative to the tokens around it.
Gap
The trivia between two tokens.
Stream
A file split for formatting.
Token
One significant token.

Functions§

line_ending
What line ending the file uses. A formatter that turned CRLF into LF would rewrite every line of a file it was asked to “only re-indent”.
stream
Split source into tokens and gaps.