fhp-tokenizer 0.1.2

SIMD-accelerated HTML tokenizer with structural indexing
Documentation

SIMD-accelerated HTML tokenizer.

Uses a two-stage pipeline inspired by simdjson:

  1. Structural indexing (SIMD): scan input in 64-byte blocks, produce per-delimiter bitmasks, then apply quote-aware masking.
  2. Token extraction (scalar): walk the structural index to emit tokens via a branchless state machine.

Quick Start

use fhp_tokenizer::tokenize;

let tokens = tokenize("<div>hello</div>");
assert!(tokens.len() >= 3);