pub struct StreamReader<'a> { /* private fields */ }Expand description
A streaming, bounded-memory parser over a (decompressing) byte source.
Build one with StreamReader::from_reader or
StreamReader::from_zstd_reader, then drive it with
par_for_each.
Use this instead of ParallelXml when the document
does not comfortably fit in memory — typically a multi-GB compressed file,
which cannot be mmap’d in its decompressed form. Resident memory is bounded
by the in-flight records (≈ threads × record size) plus one chunk,
independent of document size.
use pxml::{Event, StreamReader};
use std::sync::atomic::{AtomicUsize, Ordering};
// Any `Read` source — a File, a socket, or here a plain byte slice.
let xml = &b"<trades><trade>1</trade><trade>2</trade></trades>"[..];
let total = AtomicUsize::new(0);
StreamReader::from_reader(xml).par_for_each(|record| {
let mut events = record.events();
while let Some(ev) = events.next_event().unwrap() {
if let Event::Text(t) = ev {
total.fetch_add(t.parse::<usize>().unwrap(), Ordering::Relaxed);
}
}
})?;
assert_eq!(total.load(Ordering::Relaxed), 3);§Trade-offs vs. the resident path
Records arrive unordered and are owned (copied out of the decode
buffer rather than borrowed from it), so there is no streaming equivalent of
map_collect. In exchange memory is
constant, and on large documents throughput is often better, because the
pipeline overlaps decompression with parsing and keeps each batch
cache-resident.
Implementations§
Source§impl<'a> StreamReader<'a>
impl<'a> StreamReader<'a>
Sourcepub fn from_reader<R: Read + Send + 'a>(reader: R) -> Self
pub fn from_reader<R: Read + Send + 'a>(reader: R) -> Self
Stream over an already-decompressed byte source (any Read).
Sourcepub fn from_zstd_reader<R: Read + Send + 'a>(reader: R) -> Result<Self>
Available on crate feature zstd only.
pub fn from_zstd_reader<R: Read + Send + 'a>(reader: R) -> Result<Self>
zstd only.Stream over a zstd-compressed byte source, decompressing incrementally.
Sourcepub fn record_path<I, S>(self, path: I) -> Self
pub fn record_path<I, S>(self, path: I) -> Self
Frame the direct children of the container reached by following path,
skipping non-matching siblings — the streaming counterpart of
Config::with_record_path. Empty =
the root’s direct children (the default).
StreamReader takes no Config (the parallelism
thresholds only apply to the resident path), so the path is set here.
Sourcepub fn par_for_each<F>(self, f: F) -> Result<(), XmlError>
pub fn par_for_each<F>(self, f: F) -> Result<(), XmlError>
Frame records on a producer thread and apply f to each in parallel,
in unordered (completion) order.
Returns Err if framing or I/O fails; records already dispatched are
still processed (siblings are not aborted). Per-record parse errors are
the closure’s concern (it drives record.events()).
The workers run on rayon’s current pool. To use a specific pool, wrap the
call: pool.install(|| reader.par_for_each(f)).
Trait Implementations§
Auto Trait Implementations§
impl<'a> !RefUnwindSafe for StreamReader<'a>
impl<'a> !Sync for StreamReader<'a>
impl<'a> !UnwindSafe for StreamReader<'a>
impl<'a> Forget for StreamReader<'a>
impl<'a> Freeze for StreamReader<'a>
impl<'a> Send for StreamReader<'a>
impl<'a> Unpin for StreamReader<'a>
impl<'a> UnsafeUnpin for StreamReader<'a>
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more