panko 0.2.0

A small, zero-copy text tokenizer that crumbles strings into Words, Symbols, and Newlines.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

---
source: src/lib.rs
expression: tokens
---
[
    Symbol(
        "😀",
    ),
    Word(
        "some",
    ),
    Symbol(
        "_",
    ),
    Word(
        "str",
    ),
    Symbol(
        "😀",
    ),
]