Struct ResolvedTokenInfo 

pub struct ResolvedTokenInfo {
    pub channel: TokenChannel,
    pub token_type: TokenType,
    pub token_index: u32,
    pub start: u32,
    pub stop: u32,
    pub line: u32,
    pub column: u32,
    pub end_line: u32,
    pub end_column: u32,
    pub payload: Payload,
}

A struct with all token information usable without the TokenizedBuffer

Fields

channel: TokenChannel

Channel of the token.

token_type: TokenType

Type of the token.

token_index: u32

Token index

start: u32

Zero-based char index of the token start in the source string. Char here means a Unicode code point, not graphemes. This is what Python uses to index strings, and IDEs show for cursor position. u32 as we only support 4gb files

stop: u32

Zero-based char index of the token end in the source string. Will point to the character immediately after the token. Char here means a Unicode code point, not graphemes. This is what Python uses to index strings, and IDEs show for cursor position. u32 as we only support 4gb files

line: u32

Starting line of the token, 1-based.

column: u32

Zero-based column of the token start on the start line.

end_line: u32

Ending line of the token, 1-based.

end_column: u32

Zero-based column of the token end on the end line. This is the column of the character immediately after the token.

payload: Payload

Extra data associated with the token.

Note that the range in the string literal buffer that Payload::StringLiteral holds is a byte offset range, not a char offset range.

Trait Implementations

impl Debug for ResolvedTokenInfo

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.