Skip to main content

Crate winnow_tokio

Crate winnow_tokio 

Source
Expand description

§winnow-tokio

Crates.io Documentation CI License

Drive incremental winnow parsers from Tokio AsyncRead sources without loading the complete input into memory.

§What it does

parse_stream is a small adapter between an asynchronous byte source and a parser that understands winnow’s Partial input:

  1. Read a fixed-size chunk from an AsyncRead.
  2. Give the parser the available bytes.
  3. Read more when the parser returns ErrMode::Incomplete.
  4. Emit completed items immediately through an async callback.
  5. Drain consumed bytes before continuing.

The parser can consume input without emitting an item, which is useful for whitespace, comments, directives, and other non-semantic records. A parser that reports success without consuming input is rejected immediately so a bug cannot create an infinite loop.

§Example

use winnow::ascii::digit1;
use winnow::error::ContextError;
use winnow::Parser;
use winnow_tokio::{parse_stream, ModalResult, Partial};

fn integer_line(input: &mut Partial<&[u8]>) -> ModalResult<Option<u64>> {
    let digits: &[u8] = digit1::<_, winnow::error::ErrMode<ContextError>>
        .parse_next(input)?;
    let value = std::str::from_utf8(digits)
        .expect("digit1 only accepts ASCII digits")
        .parse()
        .expect("the digit sequence must fit in u64");
    b'\n'.parse_next(input)?;
    Ok(Some(value))
}

let input = std::io::Cursor::new(b"12\n34\n".to_vec());
let mut values = Vec::new();
winnow_tokio::parse_stream(input, integer_line, |value| {
    values.push(value);
    async { Ok::<(), std::convert::Infallible>(()) }
})
.await?;
assert_eq!(values, [12, 34]);

§Memory behavior

The feeder does not accumulate the complete stream. It retains only consumed read-ahead plus the currently incomplete item, so memory scales with the longest item rather than total input size. A single item may exceed CHUNK_SIZE; the buffer grows only as needed to finish that item.

§Scope

This crate does not define a grammar, token model, or wire format. It is a runtime adapter for applications that already use winnow for incremental parsing over Tokio byte streams.

§License

Licensed under either of:

at your option.

Structs§

Partial
Mark the input as a partial buffer for streaming input.

Enums§

Error
Errors reported by parse_stream.

Constants§

CHUNK_SIZE
Size of a single read from the underlying AsyncRead.

Functions§

parse_stream
Drive a winnow parser over a chunked AsyncRead, invoking on_item for each item the parser yields.

Type Aliases§

ModalResult
Modal error reporting for Parser::parse_next