token-parser

Parse texts into Rust data structures via a small intermediate format of nested lists and symbols (Unit).

token-parser is a backend for parsers: instead of writing a full parser for each format, you tokenize once into Units, and any type implementing Parsable can be extracted. Implementations are provided for primitives, String, Box<str>, PathBuf, Vec<T>, Box<T>, Rc<T>, Arc<T>, and (via the derive feature) for your own structs and enums.

Example

use token_parser::{Parser, Span, Unit};

let units = [Unit::Symbol("hello".into(), Span::default())];
let mut parser = Parser::new(units);
let value: String = parser.parse_next(&()).unwrap();
assert_eq!(value, "hello");

For lists:

let words = [
    Unit::Symbol("a".into(), Span::default()),
    Unit::Symbol("b".into(), Span::default()),
];
let mut parser = Parser::new([Unit::Parser(Parser::new(words))]);
let value: Vec<String> = parser.parse_next(&()).unwrap();
assert_eq!(value, vec!["a".to_string(), "b".into()]);

Features

• derive — enables #[derive(Parsable)] via token-parser-derive for structs and enums.
• radix-parsing — number parsing with configurable radix via the Context trait.

Errors

Error carries an ErrorKind (NotEnoughElements, TooManyElements, ListNotAllowed, SymbolNotAllowed, StringParsing, InvalidElement) and an optional Span.

License

MIT OR Apache-2.0