Skip to main content

CssTokenizer

Struct CssTokenizer 

Source
pub struct CssTokenizer { /* private fields */ }
Expand description

The CSS tokenizer (§4.3).

Holds the preprocessed input stream (§5.3 normalization applied at construction), the current position, and EOF tracking. Sub-algorithms are implemented as methods on this struct so they share the input stream and position.

Implementations§

Source§

impl CssTokenizer

Source

pub fn new(input: &str) -> CssTokenizer

Construct a new tokenizer over input.

Applies §5.3 input preprocessing: per the “input stream” definition, U+000D CR followed by U+000A LF is collapsed to a single LF, lone CR is replaced with LF, and U+000C FF is replaced with LF. The tokenizer operates on this normalized stream.

Trait Implementations§

Source§

impl Tokenizer for CssTokenizer

Source§

fn next_token(&mut self) -> Option<Token>

Consume and return the next token from the input stream. Read more
Source§

fn state(&self) -> State

Return the current tokenizer state.
Source§

fn set_state(&mut self, state: State)

Set the current tokenizer state. Read more
Source§

fn reset(&mut self)

Reset the tokenizer to its initial state over the same input. Read more
Source§

fn set_unicode_ranges_allowed(&mut self, allowed: bool)

§4.3.1 L782-783: Set the unicode_ranges_allowed flag. Read more
Source§

fn position(&self) -> usize

当前 position(tokenizer 内部的字符索引,0-based)。 Read more
Source§

fn next_token_with_span(&mut self) -> Option<(Token, Range<usize>)>

返回下一个 token 及其在输入流中的字符范围 [start, end)。 Read more

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.