Struct TokenizeState 

pub struct TokenizeState {
    pub max_string_len: Option<NonZeroUsize>,
    pub next_token_cannot_be_unary: bool,
    pub tokenizer_control: TokenizerControl,
    pub comment_level: usize,
    pub include_comments: bool,
    pub is_within_text_terminated_by: Option<char>,
    pub last_token: Option<SmartString<LazyCompact>>,
}

(internals) State of the tokenizer. Exported under the internals feature only.

Fields

max_string_len: Option<NonZeroUsize>

Maximum length of a string.

Not available under unchecked.

next_token_cannot_be_unary: bool

Can the next token be a unary operator?

tokenizer_control: TokenizerControl

Shared object to allow controlling the tokenizer externally.

comment_level: usize

Is the tokenizer currently inside a block comment?

include_comments: bool

Include comments?

is_within_text_terminated_by: Option<char>

Is the current tokenizer position within the text stream of an interpolated string?

last_token: Option<SmartString<LazyCompact>>

Textual syntax of the current token, if any.

Set to Some to begin tracking this information.

Trait Implementations

impl Clone for TokenizeState

fn clone(&self) -> TokenizeState

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenizeState

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TokenizeState

fn default() -> TokenizeState

Returns the “default value” for a type.

impl PartialEq for TokenizeState

fn eq(&self, other: &TokenizeState) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for TokenizeState

impl StructuralPartialEq for TokenizeState