Skip to main content

Tokenizer

Trait Tokenizer 

Source
pub trait Tokenizer {
    // Required methods
    fn next_token(&mut self) -> Option<Token>;
    fn state(&self) -> State;
    fn set_state(&mut self, state: State);
    fn reset(&mut self);
    fn set_unicode_ranges_allowed(&mut self, allowed: bool);

    // Provided methods
    fn position(&self) -> usize { ... }
    fn next_token_with_span(&mut self) -> Option<(Token, Range<usize>)> { ... }
}
Expand description

The CSS tokenizer.

Implements the tokenization algorithm of CSS Syntax §4.3. The tokenizer consumes Unicode code points from an input stream (preprocessed per §5.3 to normalize newlines) and emits Tokens.

§Algorithm structure

Unlike the HTML tokenizer’s explicit state machine, the CSS tokenizer is recursive-descent: a single entry point consume_a_token (§4.3.1) dispatches to sub-algorithms (consume_an_ident_like_token §4.3.4, consume_a_numeric_token §4.3.3, consume_a_string_token §4.3.5, consume_a_url_token §4.3.6) which in turn call primitives (consume_an_escaped_code_point §4.3.7, consume_an_ident_sequence §4.3.12, consume_a_number §4.3.13).

§Reentrancy

The CSS tokenizer is not reentrant in the same way the HTML tokenizer is: CSS has no analogue of HTML’s content-model switching (RCDATA / RAWTEXT / ScriptData). The state is therefore minimal — just “consuming” vs. “EOF emitted”. Future CSSOM incremental-parsing use cases may require reentrancy; the trait exposes state and set_state to support that future need.

Required Methods§

Source

fn next_token(&mut self) -> Option<Token>

Consume and return the next token from the input stream.

Implements the §4.3.1 “Consume a token” algorithm at the top level. Returns Some(token) for each token, ending with Some(Token::Eof) for <EOF-token> (§5.3). After <EOF-token> has been emitted, returns None.

Source

fn state(&self) -> State

Return the current tokenizer state.

Source

fn set_state(&mut self, state: State)

Set the current tokenizer state.

Currently only useful for resetting to State::Data after EOF, which re-enables token production (though the input stream is already exhausted, so this is a no-op in practice).

Source

fn reset(&mut self)

Reset the tokenizer to its initial state over the same input.

Clears any partial token state and resets position to 0. Used by future CSSOM incremental parsing and by test harnesses.

Source

fn set_unicode_ranges_allowed(&mut self, allowed: bool)

§4.3.1 L782-783: Set the unicode_ranges_allowed flag.

Defaults to false. The U+/u+ branch of §4.3.1 only produces a <unicode-range-token> (§4.3.14) when this flag is true; otherwise U/u is tokenized as an ident-like token. Per §4.3.14 L1500-1506, unicode-range tokens are not produced by the top-level tokenizer under normal circumstances — the flag is set to true only when parsing the value of the @font-face/unicode-range descriptor.

Provided Methods§

Source

fn position(&self) -> usize

当前 position(tokenizer 内部的字符索引,0-based)。

用于 next_token_with_span 追踪 token 的 source range。默认实现返回 0;具体 tokenizer 应覆盖。

注意:对 CssTokenizer 而言,这是 Vec<char> 上的 char 索引, 不是 byte offset。调用方需要自行将 char 索引映射到 byte offset。

Source

fn next_token_with_span(&mut self) -> Option<(Token, Range<usize>)>

返回下一个 token 及其在输入流中的字符范围 [start, end)。

默认实现基于 position + next_token: 记录消费前的 position 作为 start,消费后的 position 作为 end。 具体 tokenizer 可覆盖以获得更精确的 span。

返回的 range 是 char 索引(不是 byte offset),与 position 一致。EOF token 的 span 是空 range pos..pos(next_token 在 EOF 后不推进 position)。

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§