Skip to main content

tokenize

fhp_tokenizer

Function tokenize 

Source 
pub fn tokenize<'a>(input: &'a str) -> Vec<Token<'a>>
Expand description

Tokenize an HTML string into a sequence of tokens.

Convenience wrapper that runs both pipeline stages:

  1. SIMD structural indexing
  2. State-machine token extraction

ยงExample

use fhp_tokenizer::tokenize;
use fhp_tokenizer::token::Token;

let tokens = tokenize("<p>Hello &amp; world</p>");

// Should contain OpenTag, Text, CloseTag
assert!(tokens.iter().any(|t| matches!(t, Token::OpenTag { .. })));
assert!(tokens.iter().any(|t| matches!(t, Token::Text { .. })));
assert!(tokens.iter().any(|t| matches!(t, Token::CloseTag { .. })));