pub struct TokenStream {
pub tokens: Vec<Token>,
pub index: usize,
/* private fields */
}Expand description
§5.3 L1725-1754: A token stream.
Fields§
§tokens: Vec<Token>§5.3 L1730-1738: A list of tokens and/or component values. We
model component values as Tokens here for simplicity; the
§5.5 algorithms that consume “component values” wrap tokens
into crate::types::ComponentValue at the boundary.
index: usize§5.3 L1740-1748: An index into tokens, representing parsing
progress. Starts at 0. Never decreases except via
Self::restore_mark.
Implementations§
Source§impl TokenStream
impl TokenStream
Sourcepub fn new(tokens: Vec<Token>) -> Self
pub fn new(tokens: Vec<Token>) -> Self
§5.3: Construct a new token stream over tokens. The stream
implicitly appends an EOF token (§5.3 L1811-1813); we model
it by returning Token::Eof from Self::next_token when
index is out of bounds, rather than storing a sentinel.
构造的 stream 不带 source-text tracking。如需追踪原始 source
(用于 §5.5.6 original_text),请用 Self::with_source。
Sourcepub fn with_source(source: &str) -> Self
pub fn with_source(source: &str) -> Self
构造一个带 source-text tracking 的 TokenStream。
tokenize source 并记录每个 token 的 byte range,使
Self::source_slice 能返回原始 source 片段。供 §5.5.6
original_text(custom property)使用。
Sourcepub fn record_nesting_error(&self, depth: u32)
pub fn record_nesting_error(&self, depth: u32)
记录嵌套深度超限错误(仅首次覆盖,保留最早的超限点)。
Sourcepub fn nesting_error(&self) -> Option<ParseError>
pub fn nesting_error(&self) -> Option<ParseError>
返回解析过程中记录的第一个嵌套深度超限错误。
Sourcepub fn enter_nesting(&self) -> Option<u32>
pub fn enter_nesting(&self) -> Option<u32>
进入一层嵌套(consume_a_simple_block / consume_a_function
入口调用)。
返回 Some(depth) 表示该层已超过 MAX_NESTING_DEPTH:
错误已被记录,调用方应停止下降(不再递归)并返回空结果。
返回 None 表示正常进入。
Sourcepub fn leave_nesting(&self)
pub fn leave_nesting(&self)
退出一层嵌套(与 Self::enter_nesting 成对调用)。
Sourcepub fn source_slice(&self, start_index: usize, end_index: usize) -> Option<&str>
pub fn source_slice(&self, start_index: usize, end_index: usize) -> Option<&str>
返回 tokens[start_index..end_index] 对应的原始 source text。
参数是 token 索引(不是 char/byte 索引)。
返回 None 如果无 source tracking 或 index 越界。
Sourcepub fn next_token(&self) -> Token
pub fn next_token(&self) -> Token
§5.3 L1769-1773: The item of tokens at index. If
out-of-bounds, return Token::Eof.
Sourcepub fn is_empty(&self) -> bool
pub fn is_empty(&self) -> bool
§5.3 L1775-1777: A token stream is empty if the next token is
<EOF-token>.
Sourcepub fn consume_token(&mut self) -> Token
pub fn consume_token(&mut self) -> Token
§5.3 L1779-1782: Let token be the next token. Increment
index, then return token.
Sourcepub fn discard_token(&mut self)
pub fn discard_token(&mut self)
§5.3 L1784-1786: If not empty, increment index.
Sourcepub fn restore_mark(&mut self)
pub fn restore_mark(&mut self)
§5.3 L1791-1793: Pop from marked_indexes and set index to
the popped value. No-op if stack is empty (defensive).
Sourcepub fn discard_mark(&mut self)
pub fn discard_mark(&mut self)
§5.3 L1795-1797: Pop from marked_indexes and discard.
Sourcepub fn discard_whitespace(&mut self)
pub fn discard_whitespace(&mut self)
§5.3 L1799-1801: While the next token is a
<whitespace-token>, discard a token.