token-parser

Parse texts into Rust data structures via a small intermediate format of nested lists and symbols (Unit).

token-parser is a backend for parsers: instead of writing a full parser for each format, you tokenize once into Units, and any type implementing Parsable can be extracted. Implementations are provided for primitives, String, Box<str>, PathBuf, Vec<T>, Box<T>, Rc<T>, Arc<T>, and (via the derive feature) for your own structs.

Example

use token_parser::{Parser, Span, Unit};

let units = [Unit::Symbol("hello".into(), Span::default())];
let mut parser = Parser::new(units);
let value: String = parser.parse_next(&()).unwrap();
assert_eq!(value, "hello");

For lists:

let words = [
    Unit::Symbol("a".into(), Span::default()),
    Unit::Symbol("b".into(), Span::default()),
];
let mut parser = Parser::new([Unit::Parser(Parser::new(words))]);
let value: Vec<String> = parser.parse_next(&()).unwrap();
assert_eq!(value, vec!["a".to_string(), "b".into()]);

Features

• derive — enables #[derive(Parsable)] and #[derive(SymbolParsable)] for tuple, named, and unit structs (re-exported from token-parser-derive).
• radix-parsing — number parsing with configurable radix via the Context trait.

Errors

Error carries an ErrorKind (NotEnoughElements, TooManyElements, ListNotAllowed, SymbolNotAllowed, StringParsing, UnknownField, InvalidElement), an optional Span, and optional context describing what was being parsed. ErrorKind is #[non_exhaustive].

License

MIT OR Apache-2.0