Struct Token 

pub struct Token<'s> {
    pub ty: TokenType<'s>,
    pub range: Range<usize>,
    pub before_lines: Vec<TokenLine<'s>>,
    pub comments: Vec<Comment<'s>>,
    pub new_line: Option<TokenLine<'s>>,
}

A source-preserving token.

Each token contains a TokenType, defining the actual parsed token, as well as a source range and comment metadata.

Comments

Consider this token with surrounding comments:

// this is my function
/* hi */ // there
/* here it comes: */ function // do you like it?
/* some other comment */

Each comment in the source string ends up “owned” by a single token. Each token owns any comments on empty lines before it, any comments on the same line before it, and any comments on the same line after it if the token is at the end of the line. In general this is meant to match how comments are normally attached to pieces of code.

In the example above, the function token would end up looking like this:

use sqparse::token::{Comment, TerminalToken, Token, TokenLine, TokenType};
let _ = Token {
    ty: TokenType::Terminal(TerminalToken::Function),
    range: 65..73,

    before_lines: vec![
        TokenLine { comments: vec![Comment::SingleLine("this is my function")] },
        TokenLine { comments: vec![Comment::MultiLine("hi"), Comment::SingleLine("there")] },
    ],
    comments: vec![
        Comment::MultiLine("here it comes:")
    ],
    new_line: Some(TokenLine { comments: vec![Comment::SingleLine("do you like it?")] })
};

Notice how the token keeps track of the separate lines of comments before it, and whether it is followed by a newline. Also notice that the last /* some other comment */ isn’t included, since it is on a different line after the token it would be owned by whatever token comes later.

If there are un-owned comments at the end of an input when parsing completes, an Empty token will be emitted.

Fields

ty: TokenType<'s>

The type of token.

range: Range<usize>

The character range of the token in the source string.

before_lines: Vec<TokenLine<'s>>

Empty lines that appear before the token. The lines may contain comments.

comments: Vec<Comment<'s>>

Comments that appear before the token on the same line.

new_line: Option<TokenLine<'s>>

If this token ends a line, includes any comments between the token and newline.

Trait Implementations

impl<'s> Clone for Token<'s>

fn clone(&self) -> Token<'s>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'s> Debug for Token<'s>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'s> PartialEq for Token<'s>

fn eq(&self, other: &Token<'s>) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<'s> StructuralPartialEq for Token<'s>