deser_core/stream/
mod.rs

1//! Reading and writing streams of values without doing IO.
2//!
3//! Data formats parse complete inputs (slices) and serialize into complete
4//! outputs.  For streams (such as files, sockets or pipes) the formats
5//! have stream serializers and stream deserializers which do not do IO
6//! themselves (sans-io).  This makes them usable with any kind of IO, and
7//! without the standard library:
8//!
9//! * A [`StreamSerializer`](crate::ser::StreamSerializer) (the serializer
10//!   of a format, for instance `deser_json::Serializer`) holds the state
11//!   of a stream and its output.  Whoever writes the output to the stream
12//!   takes it and clears it.  Large values can be serialized in parts (see
13//!   [`drive_partial`](crate::ser::StreamSerializer::drive_partial)), so
14//!   the memory used does not depend on their size.
15//! * A [`StreamDeserializer`](crate::de::StreamDeserializer) (for instance
16//!   `deser_json::StreamDeserializer`) splits the input of a stream into
17//!   values or deserializes them in parts while their input arrives (see
18//!   [`drive_partial`](crate::de::StreamDeserializer::drive_partial)).  The
19//!   [`InputBuffer`] of this module holds the input that was read and
20//!   invokes the stream deserializer.
21//!
22//! The readers and writers of `deser::io` (which need the `io` feature)
23//! connect them to [`std::io`](https://doc.rust-lang.org/std/io/), and
24//! crates like `deser-tokio` to other kinds of IO.
25//!
26//! # Frames and Partial Deserialization
27//!
28//! A stream deserializer splits the input into frames: it finds the bytes
29//! of the next value in the input that was read so far (see
30//! [`StreamDeserializer::frame`](crate::de::StreamDeserializer::frame)),
31//! for instance a line with JSON Lines.  Once a value is complete it's
32//! deserialized from its frame with the format's regular parser.  Types
33//! can borrow from the frame (see [`InputBuffer::deserialize`]).
34//!
35//! Formats which can be parsed while the input arrives (like JSON and CBOR)
36//! can also deserialize values while the input is fed to them (see
37//! [`InputBuffer::drive_partial`]): the parts of a value are deserialized
38//! as they are read and only incomplete tokens are buffered, which means
39//! that the memory used does not depend on the size of the values.
40//!
41//! # Large Sequences
42//!
43//! Values which contain a large (or unbounded) sequence can be processed
44//! while they are read: a [`Streamed`] sequence hands out
45//! its elements as they are read with an [`ElementReader`] (and behaves
46//! like a `Vec` otherwise).
47//!
48//! # Errors
49//!
50//! Errors refer to positions in the stream: the offsets, lines and columns
51//! of errors are relative to the start of the stream, not to the start of
52//! the frame.
53//!
54//! The input ranges formats publish into the [`State`](crate::State) (and
55//! the locations derived from them, for instance by `deser-location`)
56//! refer to the frame of the value.
57mod buffer;
58pub(crate) mod elements;
59mod streamed;
60
61pub use self::buffer::{InputBuffer, Status};
62pub use self::elements::{ElementReader, ElementStatus, Part};
63pub use self::streamed::Streamed;
64
65/// The default for how much output of a value writers buffer before it's
66/// written (see `Writer::set_buffer_limit` of `deser::io`).
67///
68/// Writers pass this as the limit to
69/// [`StreamSerializer::drive_partial`](crate::ser::StreamSerializer::drive_partial).
70pub const DEFAULT_BUFFER_LIMIT: usize = 8 * 1024;
deser_core/stream/mod.rs

deser_core/stream/
mod.rs
