Struct rhai::TokenizeState
source · pub struct TokenizeState {
pub max_string_len: Option<NonZeroUsize>,
pub next_token_cannot_be_unary: bool,
pub tokenizer_control: TokenizerControl,
pub comment_level: usize,
pub include_comments: bool,
pub is_within_text_terminated_by: Option<char>,
pub last_token: Option<SmartString<LazyCompact>>,
}
Expand description
(internals) State of the tokenizer.
Exported under the internals
feature only.
Fields§
§max_string_len: Option<NonZeroUsize>
Maximum length of a string.
next_token_cannot_be_unary: bool
Can the next token be a unary operator?
tokenizer_control: TokenizerControl
Shared object to allow controlling the tokenizer externally.
comment_level: usize
Is the tokenizer currently inside a block comment?
include_comments: bool
Include comments?
is_within_text_terminated_by: Option<char>
Is the current tokenizer position within the text stream of an interpolated string?
last_token: Option<SmartString<LazyCompact>>
Textual syntax of the current token, if any.
Set to Some
to begin tracking this information.
Trait Implementations§
source§impl Clone for TokenizeState
impl Clone for TokenizeState
source§fn clone(&self) -> TokenizeState
fn clone(&self) -> TokenizeState
Returns a copy of the value. Read more
1.0.0 · source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
Performs copy-assignment from
source
. Read moresource§impl Debug for TokenizeState
impl Debug for TokenizeState
source§impl Default for TokenizeState
impl Default for TokenizeState
source§fn default() -> TokenizeState
fn default() -> TokenizeState
Returns the “default value” for a type. Read more
source§impl PartialEq<TokenizeState> for TokenizeState
impl PartialEq<TokenizeState> for TokenizeState
source§fn eq(&self, other: &TokenizeState) -> bool
fn eq(&self, other: &TokenizeState) -> bool
This method tests for
self
and other
values to be equal, and is used
by ==
.