Expand description
§winnow-tokio
Drive incremental winnow parsers from Tokio
AsyncRead
sources without loading the complete input into memory.
§What it does
parse_stream is a small adapter between an asynchronous byte source and a
parser that understands winnow’s Partial
input:
- Read a fixed-size chunk from an
AsyncRead. - Give the parser the available bytes.
- Read more when the parser returns
ErrMode::Incomplete. - Emit completed items immediately through an async callback.
- Drain consumed bytes before continuing.
The parser can consume input without emitting an item, which is useful for whitespace, comments, directives, and other non-semantic records. A parser that reports success without consuming input is rejected immediately so a bug cannot create an infinite loop.
§Example
use winnow::ascii::digit1;
use winnow::error::ContextError;
use winnow::Parser;
use winnow_tokio::{parse_stream, ModalResult, Partial};
fn integer_line(input: &mut Partial<&[u8]>) -> ModalResult<Option<u64>> {
let digits: &[u8] = digit1::<_, winnow::error::ErrMode<ContextError>>
.parse_next(input)?;
let value = std::str::from_utf8(digits)
.expect("digit1 only accepts ASCII digits")
.parse()
.expect("the digit sequence must fit in u64");
b'\n'.parse_next(input)?;
Ok(Some(value))
}
let input = std::io::Cursor::new(b"12\n34\n".to_vec());
let mut values = Vec::new();
winnow_tokio::parse_stream(input, integer_line, |value| {
values.push(value);
async { Ok::<(), std::convert::Infallible>(()) }
})
.await?;
assert_eq!(values, [12, 34]);§Memory behavior
The feeder does not accumulate the complete stream. It retains only consumed
read-ahead plus the currently incomplete item, so memory scales with the
longest item rather than total input size. A single item may exceed
CHUNK_SIZE; the buffer grows only as needed to finish that item.
§Scope
This crate does not define a grammar, token model, or wire format. It is a runtime adapter for applications that already use winnow for incremental parsing over Tokio byte streams.
§License
Licensed under either of:
- Apache License, Version 2.0 (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0)
- MIT License (LICENSE-MIT or https://opensource.org/licenses/MIT)
at your option.
Structs§
- Partial
- Mark the input as a partial buffer for streaming input.
Enums§
- Error
- Errors reported by
parse_stream.
Constants§
- CHUNK_
SIZE - Size of a single read from the underlying
AsyncRead.
Functions§
- parse_
stream - Drive a winnow parser over a chunked
AsyncRead, invokingon_itemfor each item the parser yields.