xtoken 0.1.0

Iterator based no_std XML Tokenizer using memchr
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

# xtoken


`Iterator` based `no_std` XML Tokenizer using [`memchr`](https://docs.rs/memchr).

## Design Goals


- Operates on byte slices
- Minimal Validation
- Support for inline DTD declaration
- Partition whole input into non-empty spans

## Example


```rust
use xtoken::{Token, Tokenizer};

let tokens = Tokenizer::new(b"<x>Hello World!</x>").collect::<Vec<_>>();
assert_eq!(&tokens, &[
    Token::Element(b"<x>"),
    Token::Span(b"Hello World!"),
    Token::ElementEnd(b"</x>"),
]);

let tokens = Tokenizer::new(b"<!DOCTYPE xml>").collect::<Vec<_>>();
assert_eq!(&tokens, &[
    Token::Decl(b"<!DOCTYPE xml>")
]);
```