Struct Token 

#[non_exhaustive]
pub struct Token {
    pub kind: TokenKind,
    pub value: String,
    pub line: usize,
    pub col: usize,
    pub is_quoted: bool,
    pub preceding_space: bool,
    pub preceding_whitespace: String,
    pub subst: Option<SubstPayload>,
}

A single token produced by the lexer.

Token is publicly re-exported as hocon::Token for the narrow surface that integration tests and diagnostic tooling need (per the advisory in lib.rs). It is marked #[non_exhaustive]: downstream code MUST NOT construct Token via struct-literal syntax and should treat it as inspect-only. This frees the lexer to add new metadata fields (e.g. preceding_whitespace in v1.5.3) without further source breaks.

Fields (Non-exhaustive)

Non-exhaustive structs could have additional fields added in future. Therefore, non-exhaustive structs cannot be constructed in external crates using the traditional Struct { .. } syntax; cannot be matched against without a wildcard ..; and struct update syntax will not work.

kind: TokenKind

value: String

line: usize

col: usize

is_quoted: bool

preceding_space: bool

True if preceded by whitespace OR a comment (concat detection, S10.5 / S10.8).

preceding_whitespace: String

Literal preceding-whitespace chars consumed since the previous token. Used by parse_key to preserve path-expression whitespace per E13 — for a b. c = 1 the ’ ’ before c becomes a leading-space prefix on the post-dot segment.

Note: preceding_space may be true while preceding_whitespace is empty when the token is preceded only by a comment (no literal WS chars). The boolean is the right signal for concat detection; the string is the right signal for path-WS preservation. The comment-only shape fires for the newline token emitted after // foo\n / # foo\n; non-newline tokens participating in concat / path-WS contexts are always either preceded by literal WS chars OR follow a newline that resets the buffer.

subst: Option<SubstPayload>

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.