Expand description
pxml — a parallel, StAX-style (pull) XML reader.
pxml targets one shape of document: a single root containing thousands of
uniform, order-independent records — e.g. <trades><trade>…</trade>…</trades>.
Two-phase architecture: a cheap, single-threaded boundary scan (Phase A)
frames the records and captures shared prolog context, then an
embarrassingly-parallel per-record parse (Phase B) runs on rayon. The
soundness assumption is that the framed records are independent and may be
consumed in any order.
§Quick start
Parse in parallel, collecting results back into document order:
use pxml::{Event, ParallelXml};
let xml = b"<trades><trade id='1'/><trade id='2'/></trades>".to_vec();
let doc = ParallelXml::from_bytes(xml);
let ids: Vec<String> = doc.map_collect(|record| {
let mut events = record.events();
let mut id = String::new();
while let Some(ev) = events.next_event().unwrap() {
if let Event::Start { attrs, .. } = ev {
for attr in attrs.iter() {
let attr = attr.unwrap();
if attr.key == b"id" {
id = attr.value.into_owned();
}
}
}
}
id
})?;
assert_eq!(ids, ["1", "2"]);§Choosing an entry point
| Need | Use |
|---|---|
| A file or in-memory buffer, results in document order | ParallelXml::map_collect |
| A file or in-memory buffer, order irrelevant | ParallelXml::par_for_each |
The fallible variants (closure returns Result) | ParallelXml::try_map_collect / ParallelXml::try_par_for_each |
| Record count / byte ranges only, no parsing | ParallelXml::index |
| Bounded memory over a huge or compressed stream | StreamReader |
| A classic whole-document StAX cursor | ParallelXml::sequential |
Records that are not the root’s direct children — e.g. the <object>s in
<root><manifest/><objects><object/>…</objects></root> — are reached by
setting Config::with_record_path and passing it to
ParallelXml::with_config.
§Limitations
Records must be order-independent: a worker sees only its own record, so
cross-record state (an accumulator, a reference to an earlier record) is not
available during the parallel pass. External DTDs and parameter entities are
rejected with XmlError::UnsupportedDtd; non-UTF-8 input is rejected with
XmlError::Encoding.
The design rationale is in DESIGN.md; the decisions and trade-offs actually
made are in DECISIONS.md, which supersedes it.
Structs§
- Attr
Iter - Iterator over a start tag’s
Attributes. SeeAttrs::iter. - Attribute
- One decoded attribute of a start tag.
- Attrs
- The attributes of a start tag. Iterate with
Attrs::iter(orfor); each item is anAttributewith the raw key and an entity-decoded value. - Chunk
Index - Phase A output: framing only, no parsing.
- Config
- Tuning knobs for parsing. Start from
Config::default(orConfig::new) and chain thewith_*builders, then pass the result toParallelXml::with_config. - Namespace
Context - Namespace declarations (
xmlns/xmlns:prefix) in effect for the records, captured from the root and from every element descended into on the way to the record container (seeConfig::record_path). Declarations made inside a record are local and are not stored here. - Parallel
Xml - Owns the document buffer (heap
Vecormmap) plus aConfig, and is the entry point to all parsing. - Prelude
- Immutable context shared across all workers (via
Arc). Built once in Phase A. - Record
- One top-level record: a self-contained pull reader over its slice.
- Record
Reader - A StAX-style pull cursor over one record’s events.
- SeqReader
- A sequential StAX reader over the whole document — the classic-StAX entry
point. Unlike the record API it surfaces every element (including the root
and any depth-1 text); internal-subset
<!ENTITY>definitions are captured lazily from the DOCTYPE as the document is read. - Stream
Reader - A streaming, bounded-memory parser over a (decompressing) byte source.
Enums§
- Encoding
- Resolved document encoding. The buffer is normalized to (or asserted as) UTF-8 before slicing, so workers always see UTF-8.
- Event
- A single pull-parser event, borrowed from the underlying record slice where possible (zero-copy). Comments and PIs are skipped by Phase B.
- XmlError
- Errors produced while scanning or parsing.