token-parser

Parse texts into Rust data structures via a small intermediate format of nested lists and symbols (Unit).

token-parser is a backend for parsers: instead of writing a full parser for each format, you tokenize once into Units, and any type implementing Parsable can be extracted. The reverse direction is Unparsable (value → Unit) for serialization or round-tripping. Implementations of both are provided for primitives, String, Box<str>, PathBuf, Vec<T>, Box<T>, Rc<T>, Arc<T>, and (via the derive feature) for your own structs.

Example

use token_parser::{Parser, Span, Unit};

let units = [Unit::Symbol("hello".into(), Span::default())];
let mut parser = Parser::new(units);
let value: String = parser.parse_next(&()).unwrap();
assert_eq!(value, "hello");

For lists:

let words = [
    Unit::Symbol("a".into(), Span::default()),
    Unit::Symbol("b".into(), Span::default()),
];
let mut parser = Parser::new([Unit::Parser(Parser::new(words))]);
let value: Vec<String> = parser.parse_next(&()).unwrap();
assert_eq!(value, vec!["a".to_string(), "b".into()]);

Features

• derive — enables #[derive(Parsable)] and #[derive(SymbolParsable)] for tuple, named, and unit structs (re-exported from token-parser-derive).
• radix-parsing — number parsing with configurable radix via the Context trait.

Errors

Error carries an ErrorKind (NotEnoughElements, TooManyElements, ListNotAllowed, SymbolNotAllowed, StringParsing { type_name, source }, UnknownField, InvalidElement), an optional Span (line/column plus start/end byte offsets into the source), and optional context describing what was being parsed. ErrorKind is #[non_exhaustive]. StringParsing retains the underlying FromStr error as its source.

License

MIT OR Apache-2.0