pxml
A parallel, StAX-style (pull) XML reader for Rust, built for one shape of
document: a single root containing thousands of uniform, order-independent
records — e.g. <trades><trade>…</trade>…</trades>.
pxml frames the top-level records with one cheap sequential pass, then parses
them in parallel on a rayon pool. The soundness assumption is that the
direct children of the root are independent and may be consumed in any order.
Status: v1. The full architecture from
DESIGN.mdis implemented and tested. See Limitations for the honest caveats, andDECISIONS.mdfor the design decisions, trade-offs, and benchmark analysis behind the implementation.
Why
A single linear next_event() cursor cannot be advanced by many threads — XML
events are inherently ordered and stateful. And you cannot cut the byte buffer
at an arbitrary offset and resume parsing, because a < or > may sit inside an
attribute value, comment, CDATA section, or processing instruction.
pxml resolves both problems with a two-phase, scan-then-parse design:
┌─────────────────────── whole document (Vec or mmap) ──────────────────────┐
Phase A │ <?xml?> <!DOCTYPE…> <trades> <trade>…</trade> <trade>…</trade> </trades> │
(sequential)│ └──────── prelude ────────┘ └── record 0 ──┘ └── record 1 ──┘ │
└───────────────────────────────────────────────────────────────────────────┘
│ byte ranges + shared prelude
▼
Phase B (parallel, rayon): record 0 ─▶ worker record 1 ─▶ worker …
each runs quick-xml over just its slice
- Phase A walks the buffer once with a tiny
memchr-driven state machine, finding depth-1 element boundaries and capturing shared prolog context (encoding, root namespaces, internal-subset<!ENTITY>definitions). It builds no tree and decodes no entities — it is memory-bandwidth bound. - Phase B hands each record's slice to a worker that runs a normal
quick-xmlreader over just that slice, seeded with the shared prelude so entity expansion is correct in isolation. Workers are fully independent.
Quick start
[]
= "0.2"
Requires Rust 1.88+ (edition 2024, plus let-chains used by the scanner).
use ;
use Path;
Ordered results
map_collect runs in parallel but slots results back into document order:
# use ParallelXml;
# let doc = from_bytes;
let values: = doc.map_collect?;
assert_eq!;
# Ok::
Just the framing
index() runs Phase A only — cheap, and exposes the record count and byte ranges
without parsing anything:
# use ParallelXml;
# let doc = from_bytes;
let idx = doc.index?; // Phase A only
println!;
assert_eq!; // byte range of `<r>a</r>`
# Ok::
Records under a nested container
By default the records are the root's direct children. When they instead live
inside a wrapper element — alongside siblings that should be ignored — name the
container with Config::with_record_path. pxml skips the non-matching
siblings, descends into the container (accumulating any xmlns it declares),
and frames its direct children:
# use ;
let xml = b"<root><manifest>meta</manifest>\
<objects><object/><object/></objects></root>".to_vec;
// skip <manifest>, frame each <object>
let config = new.with_record_path;
let doc = from_bytes.with_config;
assert_eq!;
# Ok::
The path may descend several levels (.with_record_path(["body", "objects"])),
and the children of every matching container are framed. An empty path (the
default) means the root itself.
The streaming reader takes no Config — the parallelism thresholds only apply
to the resident path — so it sets the path directly:
StreamReader::from_reader(r).record_path(["objects"]).
The container's namespace declarations are merged into the one shared Prelude
context (see Limitations for the multi-container caveat).
Compressed input
With the default zstd feature, from_path transparently decompresses a
zstd-compressed document (detected by its magic number); plain XML is mmap'd as
usual. For in-memory or streamed compressed data:
# // Gated so this block still compiles with --no-default-features.
#
#
The whole document is decompressed up front (workers need random access to
their slices), so decompression is sequential and adds to the serial fraction.
Build with default-features = false for a pure-Rust crate without the
C-backed zstd dependency.
Streaming (bounded memory)
from_path / from_bytes materialize the document — fine for a single file via
mmap, but a problem for a multi-GB compressed file (you can't mmap the
decompressed form), or for many large files at once. StreamReader runs the
pipeline without holding the whole document: a single producer thread
decompresses and frames records incrementally, and a rayon pool parses them in
parallel, with a bounded channel providing backpressure. Resident memory is
bounded by the in-flight records (≈ threads × record size) plus one chunk —
independent of document size.
# // Gated so this block still compiles with --no-default-features.
#
#
from_reader(impl Read) streams an already-decompressed source. Records are
framed and parsed in batches (one arena allocation each), which keeps the
producer→worker handoff cheap. The trade-offs vs. the resident path: output is
unordered and records are owned (copied out of the decode buffer rather
than borrowed). In exchange you get constant memory — and, for large documents,
often better throughput, because the pipeline overlaps decompression with
parsing and keeps each batch cache-resident instead of materializing the whole
document. On a 2M-record / 184 MiB-decompressed file the streaming path measured
~2.2× faster than from_path; see DECISIONS.md §15.
API at a glance
| Type | Purpose |
|---|---|
ParallelXml |
Owns the buffer (Vec or mmap) + Config; entry point. |
Config |
Tuning: parallel_threshold, min_records, record_path. |
ChunkIndex |
Phase A output: per-record byte ranges + shared Prelude. |
Prelude |
Immutable shared context: encoding, root name, namespaces, entities. |
StreamReader |
Bounded-memory streaming pipeline over a Read / zstd source. |
Record |
One top-level record; events() returns a pull cursor, index() its position. |
RecordReader / SeqReader |
StAX pull cursors (next_event()). |
Event |
Start { name, attrs } · End { name } · Text(Cow<str>) · Cdata(&[u8]). |
Attrs / Attribute |
Iterate a start tag's attributes (key + entity-decoded value). |
XmlError |
Malformed(pos) · Encoding · Io · UnsupportedDtd · RecordError { index, source }. |
SeqReader (via doc.sequential()) is a classic whole-document StAX reader —
the escape hatch for consumers who don't want the record model.
Pull cursors are lending:
next_event()borrows the reader, so process (or copy out of) each event before requesting the next. This keeps parsing zero-copy where possible.
Configuration & the small-input fallback
Below Config::parallel_threshold bytes or Config::min_records records,
both par_for_each and map_collect transparently run a sequential pass — the
thread-pool and indexing overhead doesn't repay itself on small inputs.
Config is built with chained with_* methods:
use ;
# let bytes = b"<rs><r>a</r></rs>".to_vec;
let config = new
.with_parallel_threshold // 1 MiB
.with_min_records;
let doc = from_bytes.with_config;
Its fields are private, so a future release can add a knob without breaking
callers. Read them back with the matching getters (config.min_records()).
Defaults: parallel_threshold = 4 MiB, min_records = 64.
Performance
Expect sub-linear scaling, not Nx. Phase A is the irreducible sequential fraction, and both phases are ultimately memory-bandwidth bound. Realistic gains are ~3–6× wall-clock on large files (hundreds of MB) with substantial per-record work, with diminishing returns past ~8 cores. Light records (small fields) bottleneck on bandwidth sooner and scale less.
Run the included benchmark (release is essential):
The in-memory mode prints a sequential baseline and par_for_each across thread
counts (throughput + speedup) plus a small-input fallback demonstration. The
file mode compares the resident from_path path against the streaming
from_zstd_reader path on a real file. See DECISIONS.md §15 for
measured numbers and analysis (notably: with batching, streaming is both
bounded-memory and ~2.2× faster than resident on a large file).
What's handled
- Encoding / BOM — UTF-8 (with or without BOM) is asserted up front; a UTF-16
BOM or a non-UTF-8 declared encoding is rejected as
XmlError::Encoding. - Namespaces —
xmlns/xmlns:prefixon the root (and on any container descended into viarecord_path) are captured into the sharedPrelude(see Limitations for how they're surfaced). - Entities — internal-subset
<!ENTITY>definitions are captured in Phase A and resolved (alongside the predefined XML entities) when decoding text and attribute values. External DTDs and parameter entities are rejected withXmlError::UnsupportedDtdrather than silently skipped. - Comments, CDATA, PIs — correctly skipped during framing, so record-lookalike text inside them never mis-frames a record. CDATA is surfaced raw; comments and PIs are not surfaced as events.
- Well-formedness — Phase A checks depth, that the root end tag's name
matches the root, and that only whitespace appears between sibling records and
descent-level elements (non-whitespace text there is rejected; content inside a
skipped sibling is not inspected). Nested-element name matching is delegated to
the per-record
quick-xmlparse; per-record parse errors carry the record'sindex. - Fallible record work —
try_par_for_each/try_map_collecttake closures returningResult, surfacing failures asXmlError::RecordError { index, .. }. - Compressed input — zstd-compressed documents are transparently
decompressed into memory (default
zstdfeature).
Limitations
v1, by design (see DESIGN.md for the full non-goals):
- Records must be order-independent. This is the load-bearing assumption: a
worker sees only its own record, so cross-record state — an accumulator, a
lookup into an earlier record, anything positional beyond
record.index()— is not available during the parallel pass. If your records are not independent, usedoc.sequential()(a classic whole-document StAX cursor) and skip the parallelism. try_map_collectdoes not short-circuit on the parallel path. Building an orderedVec<T>means visiting every record, so the closure still runs for records after a failing one, and when several fail, which error surfaces is not deterministic. Usetry_par_for_eachwhen you need early termination. (The sequential fallback does stop at the first error — so this differs between small and large inputs.)- Lexical namespaces. Element/attribute names are surfaced as written
(
QName, prefix intact). Root- and container-declared namespaces are captured inPrelude::namespacesfor manual resolution, but are not auto-applied per event.Prelude::namespacesis a single shared context, so ifrecord_pathmatches multiple containers that redeclare the same prefix to different URIs, the merge is last-writer-wins (a non-issue for uniform containers; root and ancestor declarations are always correct). - Whole document resident on the
ParallelXmlpath — workers need random access to their slices, so the document is read into aVecormmap'd. UseStreamReaderfor bounded-memory parallel parsing (at the cost of unordered, owned records). - No external DTDs / parameter entities, and no schema/DTD validation.
- Sequential Phase A. The boundary scan is single-threaded (a speculative parallel scan is a possible future optimization).
- Parallelism is at the record level — the root's direct children by default, or
the children of a container named via
record_path. Content nested within a record is parsed sequentially (fine for the uniform-records target).
Development
Built on quick-xml (Phase B parsing), rayon (work-stealing pool),
memchr (delimiter scanning), memmap2 (zero-copy file mapping), and
zstd (optional decompression).
License
Licensed under the MIT License.