Skip to main content

StreamReader

Struct StreamReader 

Source
pub struct StreamReader<'a> { /* private fields */ }
Expand description

A streaming, bounded-memory parser over a (decompressing) byte source.

Build one with StreamReader::from_reader or StreamReader::from_zstd_reader, then drive it with par_for_each.

Use this instead of ParallelXml when the document does not comfortably fit in memory — typically a multi-GB compressed file, which cannot be mmap’d in its decompressed form. Resident memory is bounded by the in-flight records (≈ threads × record size) plus one chunk, independent of document size.

use pxml::{Event, StreamReader};
use std::sync::atomic::{AtomicUsize, Ordering};

// Any `Read` source — a File, a socket, or here a plain byte slice.
let xml = &b"<trades><trade>1</trade><trade>2</trade></trades>"[..];
let total = AtomicUsize::new(0);

StreamReader::from_reader(xml).par_for_each(|record| {
    let mut events = record.events();
    while let Some(ev) = events.next_event().unwrap() {
        if let Event::Text(t) = ev {
            total.fetch_add(t.parse::<usize>().unwrap(), Ordering::Relaxed);
        }
    }
})?;

assert_eq!(total.load(Ordering::Relaxed), 3);

§Trade-offs vs. the resident path

Records arrive unordered and are owned (copied out of the decode buffer rather than borrowed from it), so there is no streaming equivalent of map_collect. In exchange memory is constant, and on large documents throughput is often better, because the pipeline overlaps decompression with parsing and keeps each batch cache-resident.

Implementations§

Source§

impl<'a> StreamReader<'a>

Source

pub fn from_reader<R: Read + Send + 'a>(reader: R) -> Self

Stream over an already-decompressed byte source (any Read).

Source

pub fn from_zstd_reader<R: Read + Send + 'a>(reader: R) -> Result<Self>

Available on crate feature zstd only.

Stream over a zstd-compressed byte source, decompressing incrementally.

Source

pub fn record_path<I, S>(self, path: I) -> Self
where I: IntoIterator<Item = S>, S: Into<Box<str>>,

Frame the direct children of the container reached by following path, skipping non-matching siblings — the streaming counterpart of Config::with_record_path. Empty = the root’s direct children (the default).

StreamReader takes no Config (the parallelism thresholds only apply to the resident path), so the path is set here.

Source

pub fn par_for_each<F>(self, f: F) -> Result<(), XmlError>
where F: Fn(&Record<'_>) + Sync,

Frame records on a producer thread and apply f to each in parallel, in unordered (completion) order.

Returns Err if framing or I/O fails; records already dispatched are still processed (siblings are not aborted). Per-record parse errors are the closure’s concern (it drives record.events()).

The workers run on rayon’s current pool. To use a specific pool, wrap the call: pool.install(|| reader.par_for_each(f)).

Trait Implementations§

Source§

impl Debug for StreamReader<'_>

Source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more

Auto Trait Implementations§

§

impl<'a> !RefUnwindSafe for StreamReader<'a>

§

impl<'a> !Sync for StreamReader<'a>

§

impl<'a> !UnwindSafe for StreamReader<'a>

§

impl<'a> Forget for StreamReader<'a>

§

impl<'a> Freeze for StreamReader<'a>

§

impl<'a> Send for StreamReader<'a>

§

impl<'a> Unpin for StreamReader<'a>

§

impl<'a> UnsafeUnpin for StreamReader<'a>

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> IntoEither for T

Source§

fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ

Converts self into a Left variant of Either<Self, Self> if into_left is true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
where F: FnOnce(&Self) -> bool,

Converts self into a Left variant of Either<Self, Self> if into_left(&self) returns true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

impl<T> Pointable for T

Source§

const ALIGN: usize

The alignment of pointer.
Source§

type Init = T

The type for initializers.
Source§

unsafe fn init(init: <T as Pointable>::Init) -> usize

Initializes a with the given initializer. Read more
Source§

unsafe fn deref<'a>(ptr: usize) -> &'a T

Dereferences the given pointer. Read more
Source§

unsafe fn deref_mut<'a>(ptr: usize) -> &'a mut T

Mutably dereferences the given pointer. Read more
Source§

unsafe fn drop(ptr: usize)

Drops the object pointed to by the given pointer. Read more
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.