Crate html5tokenizer

source ·
Expand description


Spec-compliant HTML parsing requires both tokenization and tree-construction. While this crate implements a spec-compliant HTML tokenizer it does not implement any tree-construction. Instead it just provides a NaiveParser that may be used as follows:

use std::fmt::Write;
use html5tokenizer::{NaiveParser, Token};

let html = "<title   >hello world</title>";
let mut new_html = String::new();

for token in NaiveParser::new(html).flatten() {
    match token {
        Token::StartTag(tag) => {
            write!(new_html, "<{}>",;
        Token::Char(c) => {
            write!(new_html, "{c}").unwrap();
        Token::EndTag(tag) => {
            write!(new_html, "</{}>",;
        Token::EndOfFile => {},
        _ => panic!("unexpected input"),

assert_eq!(new_html, "<title>hello world</title>");

This library can provide source spans. For an example, see examples/, which produces the following output:

  ┌─ file.html:1:2
1 │ <img src=example.jpg alt="some description">
  │  ^^^ ^^^ ^^^^^^^^^^^ ^^^  ^^^^^^^^^^^^^^^^ attr value
  │  │   │   │           │
  │  │   │   │           attr name
  │  │   │   attr value
  │  │   attr name
  │  tag name


  • This crate does not yet implement tree construction
    (which is necessary for spec-compliant HTML parsing).

  • This crate does not yet implement character encoding detection.

Compliance & testing

The tokenizer passes the html5lib tokenizer test suite. The library is not yet fuzz tested.


html5tokenizer was forked from html5gum 0.2.1, which was created by Markus Unterwaditzer who deserves major props for implementing all 80 (!) tokenizer states.

  • Code span support has been added.
  • The API has been revised.

For details please refer to the changelog.


Licensed under the MIT license, see the LICENSE file.



  • Types for HTML attributes.
  • Source code offsets.
  • Provides the Reader trait (and implementations) used by the tokenizer.
  • Provides the Token type.
  • Provides the Trace type (byte offsets and syntax information about tokens).




  • An emitter is an object providing methods to the tokenizer to produce (“emit”) tokens.