winnow/_topic/partial.rs
1//! # Parsing Partial Input
2//!
3//! Typically, the input being parsed is all in-memory, or is complete. Some data sources are too
4//! large to fit into memory, only allowing parsing an incomplete or [`Partial`] subset of the
5//! data, requiring incrementally parsing.
6//!
7//! By wrapping a stream, like `&[u8]`, with [`Partial`], parsers will report when the data is
8//! [`Incomplete`] and more input is [`Needed`], allowing the caller to stream-in additional data
9//! to be parsed. The data is then parsed a chunk at a time.
10//!
11//! Chunks are typically defined by either:
12//! - A header reporting the number of bytes, like with [`length_and_then`]
13//! - [`Partial`] can explicitly be changed to being complete once the specified bytes are
14//! acquired via [`StreamIsPartial::complete`].
15//! - A delimiter, like with [ndjson](https://github.com/ndjson/ndjson-spec/)
16//! - You can parse up-to the delimiter or do a `take_until(0.., delim).and_then(parser)`
17//!
18//! If the chunks are not homogeneous, a state machine will be needed to track what the expected
19//! parser is for the next chunk.
20//!
21//! Caveats:
22//! - `winnow` takes the approach of re-parsing from scratch. Chunks should be relatively small to
23//! prevent the re-parsing overhead from dominating.
24//! - Parsers like [`repeat`] do not know when an `eof` is from insufficient data or the end of the
25//! stream, causing them to always report [`Incomplete`].
26//!
27//! # Example
28//!
29//! `main.rs`:
30//! ```rust,ignore
31#![doc = include_str!("../../examples/ndjson/main.rs")]
32//! ```
33//!
34//! `parser.rs`:
35//! ```rust,ignore
36#![doc = include_str!("../../examples/ndjson/parser.rs")]
37//! ```
38
39#![allow(unused_imports)] // Used for intra-doc links
40
41use crate::binary::length_and_then;
42use crate::combinator::repeat;
43use crate::error::ErrMode::Incomplete;
44use crate::error::Needed;
45use crate::stream::Partial;
46use crate::stream::StreamIsPartial;