Skip to main content

Tokenizer

Trait Tokenizer 

Source
pub trait Tokenizer {
    // Required methods
    fn next_token(&mut self) -> Option<Token>;
    fn state(&self) -> State;
    fn set_state(&mut self, state: State);
    fn reset(&mut self);
    fn set_unicode_ranges_allowed(&mut self, allowed: bool);
}
Expand description

The CSS tokenizer.

Implements the tokenization algorithm of CSS Syntax §4.3. The tokenizer consumes Unicode code points from an input stream (preprocessed per §5.3 to normalize newlines) and emits Tokens.

§Algorithm structure

Unlike the HTML tokenizer’s explicit state machine, the CSS tokenizer is recursive-descent: a single entry point consume_a_token (§4.3.1) dispatches to sub-algorithms (consume_an_ident_like_token §4.3.4, consume_a_numeric_token §4.3.3, consume_a_string_token §4.3.5, consume_a_url_token §4.3.6) which in turn call primitives (consume_an_escaped_code_point §4.3.7, consume_an_ident_sequence §4.3.12, consume_a_number §4.3.13).

§Reentrancy

The CSS tokenizer is not reentrant in the same way the HTML tokenizer is: CSS has no analogue of HTML’s content-model switching (RCDATA / RAWTEXT / ScriptData). The state is therefore minimal — just “consuming” vs. “EOF emitted”. Future CSSOM incremental-parsing use cases may require reentrancy; the trait exposes state and set_state to support that future need.

Required Methods§

Source

fn next_token(&mut self) -> Option<Token>

Consume and return the next token from the input stream.

Implements the §4.3.1 “Consume a token” algorithm at the top level. Returns Some(token) for each token, ending with Some(Token::Eof) for <EOF-token> (§5.3). After <EOF-token> has been emitted, returns None.

Source

fn state(&self) -> State

Return the current tokenizer state.

Source

fn set_state(&mut self, state: State)

Set the current tokenizer state.

Currently only useful for resetting to State::Data after EOF, which re-enables token production (though the input stream is already exhausted, so this is a no-op in practice).

Source

fn reset(&mut self)

Reset the tokenizer to its initial state over the same input.

Clears any partial token state and resets position to 0. Used by future CSSOM incremental parsing and by test harnesses.

Source

fn set_unicode_ranges_allowed(&mut self, allowed: bool)

§4.3.1 L782-783: Set the unicode_ranges_allowed flag.

Defaults to false. The U+/u+ branch of §4.3.1 only produces a <unicode-range-token> (§4.3.14) when this flag is true; otherwise U/u is tokenized as an ident-like token. Per §4.3.14 L1500-1506, unicode-range tokens are not produced by the top-level tokenizer under normal circumstances — the flag is set to true only when parsing the value of the @font-face/unicode-range descriptor.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§