Skip to main content

Crate pxml

Crate pxml 

Source
Expand description

pxml — a parallel, StAX-style (pull) XML reader.

pxml targets one shape of document: a single root containing thousands of uniform, order-independent records — e.g. <trades><trade>…</trade>…</trades>.

Two-phase architecture: a cheap, single-threaded boundary scan (Phase A) frames the records and captures shared prolog context, then an embarrassingly-parallel per-record parse (Phase B) runs on rayon. The soundness assumption is that the framed records are independent and may be consumed in any order.

§Quick start

Parse in parallel, collecting results back into document order:

use pxml::{Event, ParallelXml};

let xml = b"<trades><trade id='1'/><trade id='2'/></trades>".to_vec();
let doc = ParallelXml::from_bytes(xml);

let ids: Vec<String> = doc.map_collect(|record| {
    let mut events = record.events();
    let mut id = String::new();
    while let Some(ev) = events.next_event().unwrap() {
        if let Event::Start { attrs, .. } = ev {
            for attr in attrs.iter() {
                let attr = attr.unwrap();
                if attr.key == b"id" {
                    id = attr.value.into_owned();
                }
            }
        }
    }
    id
})?;

assert_eq!(ids, ["1", "2"]);

§Choosing an entry point

NeedUse
A file or in-memory buffer, results in document orderParallelXml::map_collect
A file or in-memory buffer, order irrelevantParallelXml::par_for_each
The fallible variants (closure returns Result)ParallelXml::try_map_collect / ParallelXml::try_par_for_each
Record count / byte ranges only, no parsingParallelXml::index
Bounded memory over a huge or compressed streamStreamReader
A classic whole-document StAX cursorParallelXml::sequential

Records that are not the root’s direct children — e.g. the <object>s in <root><manifest/><objects><object/>…</objects></root> — are reached by setting Config::with_record_path and passing it to ParallelXml::with_config.

§Limitations

Records must be order-independent: a worker sees only its own record, so cross-record state (an accumulator, a reference to an earlier record) is not available during the parallel pass. External DTDs and parameter entities are rejected with XmlError::UnsupportedDtd; non-UTF-8 input is rejected with XmlError::Encoding.

The design rationale is in DESIGN.md; the decisions and trade-offs actually made are in DECISIONS.md, which supersedes it.

Structs§

AttrIter
Iterator over a start tag’s Attributes. See Attrs::iter.
Attribute
One decoded attribute of a start tag.
Attrs
The attributes of a start tag. Iterate with Attrs::iter (or for); each item is an Attribute with the raw key and an entity-decoded value.
ChunkIndex
Phase A output: framing only, no parsing.
Config
Tuning knobs for parsing. Start from Config::default (or Config::new) and chain the with_* builders, then pass the result to ParallelXml::with_config.
NamespaceContext
Namespace declarations (xmlns / xmlns:prefix) in effect for the records, captured from the root and from every element descended into on the way to the record container (see Config::record_path). Declarations made inside a record are local and are not stored here.
ParallelXml
Owns the document buffer (heap Vec or mmap) plus a Config, and is the entry point to all parsing.
Prelude
Immutable context shared across all workers (via Arc). Built once in Phase A.
Record
One top-level record: a self-contained pull reader over its slice.
RecordReader
A StAX-style pull cursor over one record’s events.
SeqReader
A sequential StAX reader over the whole document — the classic-StAX entry point. Unlike the record API it surfaces every element (including the root and any depth-1 text); internal-subset <!ENTITY> definitions are captured lazily from the DOCTYPE as the document is read.
StreamReader
A streaming, bounded-memory parser over a (decompressing) byte source.

Enums§

Encoding
Resolved document encoding. The buffer is normalized to (or asserted as) UTF-8 before slicing, so workers always see UTF-8.
Event
A single pull-parser event, borrowed from the underlying record slice where possible (zero-copy). Comments and PIs are skipped by Phase B.
XmlError
Errors produced while scanning or parsing.