SIMD-accelerated HTML tokenizer.
Uses a two-stage pipeline inspired by simdjson:
- Structural indexing (SIMD): scan input in 64-byte blocks, produce per-delimiter bitmasks, then apply quote-aware masking.
- Token extraction (scalar): walk the structural index to emit tokens via a branchless state machine.
Quick Start
use tokenize;
let tokens = tokenize;
assert!;