Skip to main content

StreamReader

Struct StreamReader 

Source
pub struct StreamReader { /* private fields */ }
Expand description

A pull reader over input that arrives in pieces: bytes go in with feed as they arrive, events come out as soon as they are complete.

It is the same reader as Reader, over a buffer: the same grammar, the same budgets, and the same errors at the same offsets (counted from the first byte ever fed), whatever the input’s chunking. When a call needs bytes that have not arrived it returns Step::NeedInput and changes nothing a later call would see differently; after end_input, running out is the ErrorKind::UnexpectedEnd that Reader reports.

Events borrow from the reader’s buffer, so each one must be dropped before the next call. In particular, feed more input after the borrow ends: in an else after if let Step::Ready(..) = reader.next_event()? (Rust 2024 drops the if let temporaries before the else), or after the loop that drains the ready events, as below; a match keeps the event borrowed in every arm. Consumed input is dropped from the buffer as more is fed, so a long document read with next_event as it arrives needs memory for one token and one chunk, not for the document. The whole-value calls (read_raw, read_value, decode) are different: each keeps its value’s text in the buffer until the value ends, bounded by Limits::max_value_bytes. A token cut by a chunk boundary is continued where it stopped, never scanned again from its start: a string fed one byte at a time costs linear time.

The input must be UTF-8, checked as it arrives (a character may be split across chunks). The first invalid byte is reported as ErrorKind::InvalidUtf8 when the reader reaches it: a syntax error earlier in the document is reported first. (A complete document given to parse_bytes checks the encoding before the syntax, so the two can name different errors for a document that has both.)

Limits::max_input_bytes bounds the total fed (in documents mode, what is fed for each document), and Limits::max_value_bytes what a whole-value call may hold; the other budgets apply exactly as for a complete document. After an error the reader is finished: every later call returns the same error.

§Examples

use jsonx_lib::{Event, Step, StreamReader};

let mut reader = StreamReader::new();
let mut keys = Vec::new();
for chunk in [&b"{\"jsonr"[..], b"pc\": \"2.0\", \"i", b"d\": 12", b"}"] {
    reader.feed(chunk)?;
    while let Step::Ready(Some(event)) = reader.next_event()? {
        if let Event::Key(key) = event {
            keys.push(key.into_owned());
        }
    }
}
reader.end_input();
assert_eq!(reader.next_event()?, Step::Ready(None)); // the document is complete
assert_eq!(keys, ["jsonrpc", "id"]);

Implementations§

Source§

impl StreamReader

Source

pub fn new() -> Self

A reader with the default Limits, refusing duplicate keys in what read_value builds. For others, use Parser::stream_reader.

§Examples
let r = jsonx_lib::StreamReader::new();
assert_eq!(r.offset(), 0);
Source

pub fn documents(self) -> Self

Reads a sequence of documents instead of one, as Reader::documents does: NDJSON, JSON Lines, or concatenated JSON, fed as it arrives. Whole-value reads return one document each, and Ready(None) once the input has ended after the last.

Limits::max_input_bytes then applies to each document, not to the stream: input is counted from where the current document began (or, between documents, from the end of the last one), so a connection that carries documents for as long as it stays open is never refused for its total. A Reader, which has its whole input already, applies the budget to all of it.

§Examples
use jsonx_lib::{Step, StreamReader};

let mut r = StreamReader::new().documents();
let mut seen = Vec::new();
for chunk in [&b"{\"id\": 1}\n{\"i"[..], b"d\": 2}\n"] {
    r.feed(chunk)?;
    while let Step::Ready(Some(doc)) = r.read_value()? {
        seen.push(doc["id"].as_i64());
    }
}
r.end_input();
assert_eq!(r.read_value()?, Step::Ready(None));
assert_eq!(seen, [Some(1), Some(2)]);
Source

pub fn feed(&mut self, bytes: &[u8]) -> Result<(), Error>

Adds the next piece of input. A chunk may end anywhere, even inside a token or a UTF-8 character.

§Errors

ErrorKind::InputTooLarge when the input in total (in documents mode, of the current document) grows past Limits::max_input_bytes, or when the value a pending whole-value call is reading grows past Limits::max_value_bytes, at the offset of the first byte over the budget (the reader is then finished); ErrorKind::InputAfterEnd after end_input; any error the reader already stopped at.

§Examples
use jsonx_lib::{ErrorKind, Limits, Parser};

let mut r = Parser::new().limits(Limits::new().with_max_input_bytes(4)).stream_reader();
assert!(r.feed(b"[1,").is_ok());
assert_eq!(r.feed(b"22]").unwrap_err().kind(), ErrorKind::InputTooLarge);
Source

pub fn end_input(&mut self)

Says that the input has ended: nothing more will be fed. From now on, running out of input is an error, as for a complete document, and a number or whitespace at the very end is final.

§Examples
use jsonx_lib::{Event, Step, StreamReader};

let mut r = StreamReader::new();
r.feed(b"12")?;
assert_eq!(r.next_event()?, Step::NeedInput); // `123` may follow
r.end_input();
assert_eq!(r.next_event()?, Step::Ready(Some(Event::Number(12.into()))));
assert_eq!(r.next_event()?, Step::Ready(None));
Source

pub fn read_from(&mut self, source: &mut dyn Read) -> Result<usize>

Available on crate feature std only.

Reads up to 8 KiB from source and feeds it, or calls end_input when source is at its end. Returns the number of bytes read (0 at the end). Interrupted reads are retried.

A budget error from feed is not an I/O error: it is kept, and returned by the next reading call.

§Errors

Any error from source other than std::io::ErrorKind::Interrupted.

§Examples
use jsonx_lib::{Step, StreamReader};

let mut source: &[u8] = br#"{"big": [1, 2, 3]}"#;
let mut r = StreamReader::new();
let mut events = 0;
loop {
    if let Step::Ready(event) = r.next_event()? {
        match event {
            Some(_) => events += 1,
            None => break,
        }
    } else {
        r.read_from(&mut source)?;
    }
}
assert_eq!(events, 8); // {, "big", [, 1, 2, 3, ], }
Source

pub fn next_event(&mut self) -> Result<Step<Option<Event<'_>>>, Error>

Reads the next event: Ready(Some(event)), Ready(None) once the document is complete and the input has ended with nothing but whitespace after it, or NeedInput.

§Errors

The first syntax error, exceeded budget, or invalid UTF-8, at its byte offset. Once an error is returned, every later call returns it.

§Examples
use jsonx_lib::{ErrorKind, Event, Step, StreamReader};

let mut r = StreamReader::new();
r.feed(b"[1,")?;
assert_eq!(r.next_event()?, Step::Ready(Some(Event::StartArray)));
assert!(r.next_event()?.is_ready()); // 1
r.feed(b"]")?;
let err = r.next_event().unwrap_err();
assert_eq!((err.kind(), err.offset()), (ErrorKind::TrailingComma, 3));
Source

pub fn read_raw(&mut self) -> Result<Step<Option<&str>>, Error>

Reads the next value whole and returns its exact source text, without decoding or allocating anything, as Reader::read_raw does, with the same positioning rules: Ready(None) when no value comes next.

The value must be in the buffer whole before it can be returned, so a large value costs memory for all of it, whitespace included, up to Limits::max_value_bytes (past it, feed refuses more). NeedInput keeps the progress: the next call continues the same value.

After a single document’s value, a call answers as next_event does there: NeedInput while only whitespace has followed and the input has not ended, Ready(None) once it has, and ErrorKind::TrailingCharacters for anything but whitespace. finish makes the same check in one call.

§Errors

As next_event, for any error inside the value, and ErrorKind::TrailingCharacters after the document.

§Examples
use jsonx_lib::{Event, Step, StreamReader};

let mut r = StreamReader::new();
r.feed(br#"{"id": 1, "params": {"line""#)?;
assert!(r.next_event()?.is_ready()); // {
assert!(r.next_event()?.is_ready()); // "id"
assert_eq!(r.read_raw()?, Step::Ready(Some("1")));
assert!(r.next_event()?.is_ready()); // "params"
assert_eq!(r.read_raw()?, Step::NeedInput);
r.feed(br#": 3}}"#)?;
assert_eq!(r.read_raw()?, Step::Ready(Some(r#"{"line": 3}"#)));
Source

pub fn read_value(&mut self) -> Result<Step<Option<Value<'_>>>, Error>

Reads the next value whole and builds it as a Value, applying the reader’s DuplicateKeys policy; positioning as for read_raw. The value borrows from the buffer: call Value::into_owned to keep it.

§Errors

As read_raw, plus ErrorKind::DuplicateKey under DuplicateKeys::Error.

§Examples
use jsonx_lib::{Step, StreamReader};

let mut r = StreamReader::new();
r.feed(br#"{"a": [1, "#)?;
assert_eq!(r.read_value()?, Step::NeedInput);
r.feed(br#"2]}"#)?;
let value = r.read_value()?.ready().flatten().unwrap_or_default();
assert_eq!(value["a"][1], 2);
Source

pub fn decode<'s, T: FromJson<'s>>( &'s mut self, ) -> Result<Step<Option<T>>, DecodeError>

Reads the next value whole as a T (FromJson); positioning as for read_raw. The value borrows from the buffer, as T allows.

The value is first skipped (validated, with every budget) as it arrives, then read as T once it is all there. Byte offsets in an error count from the first byte fed.

§Errors

The DecodeError of T, or the reader’s error.

§Examples
use jsonx_lib::{Step, StreamReader};

let mut r = StreamReader::new().documents();
r.feed(b"[1, 2]\n[3")?;
assert_eq!(r.decode::<Vec<u8>>()?, Step::Ready(Some(vec![1, 2])));
assert_eq!(r.decode::<Vec<u8>>()?, Step::NeedInput);
r.feed(b"]\n")?;
r.end_input();
assert_eq!(r.decode::<Vec<u8>>()?, Step::Ready(Some(vec![3])));
assert_eq!(r.decode::<Vec<u8>>()?, Step::Ready(None));
Source

pub fn finish(&mut self) -> Result<Step<()>, Error>

Reads to the end of the input, checking that the rest of the document (in documents mode, of every document) is valid and that nothing but whitespace follows: Ready(()) once the input has ended and all of it was read, NeedInput before that. As Reader::finish, for input that arrives in pieces.

The whole-value calls stop at the end of the value they read, and a value nested in the document says nothing of the rest of it. Call this after the last of them to know that the input held exactly one valid document (or a sequence of valid ones).

§Errors

The first error in the rest of the input, as next_event reports it: a syntax error, an exceeded budget, invalid UTF-8, or ErrorKind::TrailingCharacters.

§Examples
use jsonx_lib::{ErrorKind, Step, StreamReader};

let mut r = StreamReader::new();
r.feed(br#"{"id": 1} "#)?;
assert!(r.read_value()?.is_ready());
assert_eq!(r.finish()?, Step::NeedInput); // more may follow
r.end_input();
assert_eq!(r.finish()?, Step::Ready(()));

let mut r = StreamReader::new();
r.feed(br#"{"id": 1}}"#)?;
r.end_input();
assert!(r.read_value()?.is_ready());
let err = r.finish().unwrap_err();
assert_eq!((err.kind(), err.offset()), (ErrorKind::TrailingCharacters, 9));
Source

pub fn peek_kind(&mut self) -> Result<Step<Option<Kind>>, Error>

The kind of the next value, without consuming anything: Ready(None) when no value comes next, or when the next byte cannot start one (the next next_event reports why).

§Errors

Any error the reader already stopped at.

§Examples
use jsonx_lib::{Kind, Step, StreamReader};

let mut r = StreamReader::new();
r.feed(b"[ ")?;
assert!(r.next_event()?.is_ready());
assert_eq!(r.peek_kind()?, Step::NeedInput);
r.feed(b"\"x\"]")?;
assert_eq!(r.peek_kind()?, Step::Ready(Some(Kind::String)));
Source

pub fn depth(&self) -> usize

The number of arrays and objects open.

§Examples
let mut r = jsonx_lib::StreamReader::new();
r.feed(b"[[")?;
let _ = r.next_event()?;
let _ = r.next_event()?;
assert_eq!(r.depth(), 2);
Source

pub fn offset(&self) -> usize

The byte offset, from the first byte fed, just past the input consumed so far.

§Examples
let mut r = jsonx_lib::StreamReader::new();
r.feed(b"[10, 20]")?;
let _ = r.next_event()?;
let _ = r.next_event()?;
assert_eq!(r.offset(), 3);
Source

pub fn event_offset(&self) -> usize

The byte offset, from the first byte fed, where the most recent event’s token began (see Reader::event_offset).

§Examples
let mut r = jsonx_lib::StreamReader::new();
r.feed(b"[ 7]")?;
let _ = r.next_event()?;
let _ = r.next_event()?;
assert_eq!(r.event_offset(), 2);
Source

pub fn buffered(&self) -> usize

The bytes of input the reader holds: what has been fed and not yet dropped. Consumed input is dropped as more is fed.

§Examples
let mut r = jsonx_lib::StreamReader::new();
r.feed(b"[1, 2")?;
assert_eq!(r.buffered(), 5);

Trait Implementations§

Source§

impl Clone for StreamReader

Source§

fn clone(&self) -> Self

Returns a duplicate of the value. Read more
1.0.0 (const: unstable) · Source§

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source. Read more
Source§

impl Debug for StreamReader

Source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more
Source§

impl Default for StreamReader

Source§

fn default() -> Self

Returns the “default value” for a type. Read more

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> CloneToUninit for T
where T: Clone,

Source§

unsafe fn clone_to_uninit(&self, dest: *mut u8)

🔬This is a nightly-only experimental API. (clone_to_uninit)
Performs copy-assignment from self to dest. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> ToOwned for T
where T: Clone,

Source§

type Owned = T

The resulting type after obtaining ownership.
Source§

fn to_owned(&self) -> T

Creates owned data from borrowed data, usually by cloning. Read more
Source§

fn clone_into(&self, target: &mut T)

Uses borrowed data to replace owned data, usually by cloning. Read more
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.