muskitty-css-tokenizer
A from-scratch CSS tokenizer written in pure Rust, implementing the CSS Syntax Module Level 3 §4.3 with zero runtime dependencies.
Part of the MusKitty browser engine project.
Status
| Component | Spec Coverage | Test Pass Rate |
|---|---|---|
| Tokenizer (§4.3) | 15/15 sub-algorithms | 71/71 unit tests |
- Zero
unsafecode - Zero C/C++ dependencies
- Zero runtime dependencies
- Rust stable toolchain only
- MSRV 1.82
Installation
Add this to your Cargo.toml:
[]
= "0.1.0"
Or run:
Quick Start
use ;
let mut t = new;
while let Some = t.next_token
Architecture
muskitty-css-tokenizer/
src/
types.rs Token, HashType, Numeric, State definitions
trait_def.rs Tokenizer trait
impls.rs CssTokenizer — recursive-descent tokenizer (~1800 lines)
lib.rs Public API: CssTokenizer + Tokenizer trait + types
What is a CSS Tokenizer?
The CSS tokenizer is a recursive-descent algorithm (§4.3) that consumes
a stream of Unicode code points (after §5.3 preprocessing — CR/LF/FF
normalized to LF) and emits tokens. The main entry point
consume_a_token (§4.3.1) dispatches to sub-algorithms based on the
current input code point.
Spec Coverage
All 15 §4.3 sub-algorithms are implemented:
- §4.3.1 Consume a token (full dispatch, incl.
unicode_ranges_allowed) - §4.3.2 Consume comments
- §4.3.3 Consume a numeric token
- §4.3.4 Consume an ident-like token (incl.
url(special case) - §4.3.5 Consume a string token
- §4.3.6 Consume a url token
- §4.3.7 Consume an escaped code point
- §4.3.8 Check if two code points are a valid escape
- §4.3.9 Check if three code points would start an ident sequence
- §4.3.10 Check if three code points would start a number
- §4.3.11 Check if three code points would start a unicode-range
- §4.3.12 Consume an ident sequence
- §4.3.13 Consume a number
- §4.3.14 Consume a unicode-range token
- §4.3.15 Consume the remnants of a bad url
Building
Testing
# Unit tests (71 tests)
# All tests
Design Principles
- CSSWG is ground truth — Implementation follows the spec exactly.
- Spec-compliant, not test-compliant — Tests verify the code; code is never modified to pass a test unless the spec proves the test is wrong.
- Zero runtime dependencies — Pure safe Rust.
- Zero unsafe — Pure safe Rust.
- Surgical changes — Every diff is as small as the task requires.
Spec Reference
This implementation references:
- CSS Syntax Module Level 3 — Primary authority
- §4.1: Token Railroad Diagrams
- §4.3: Tokenizer Algorithms
- §5.3: Input Stream Preprocessing
License
Licensed under the Apache License, Version 2.0. See LICENSE for details.
Copyright 2026 MusCat / MusKitty Bit-Torch Community