Crate regex_chunker 

The centerpiece of this crate is the ByteChunker, which takes a regular expression and wraps a Read type, becoming an iterator over the bytes read from the wrapped type, yielding chunks delimited by the supplied regular expression.

The example program below uses a ByteChunker to do a crude word tally on text coming in on the standard input.

use std::{collections::BTreeMap, error::Error};
use regex_chunker::ByteChunker;
  
fn main() -> Result<(), Box<dyn Error>> {
    let mut counts: BTreeMap<String, usize> = BTreeMap::new();
    let stdin = std::io::stdin();
    
    // The regex is a stab at something matching strings of
    // "between-word" characters in general English text.
    let chunker = ByteChunker::new(stdin, r#"[ "\r\n.,!?:;/]+"#)?;
    for chunk in chunker {
        let word = String::from_utf8_lossy(&chunk?).to_lowercase();
        *counts.entry(word).or_default() += 1;
    }

    println!("{:#?}", &counts);
    Ok(())
}

Enabling the async feature also exposes the stream module, which features an async version of the ByteChunker, wrapping an AsyncRead and implementing Stream.

(This also pulls in several crates of tokio machinery, which is why it’s behind a feature flag.)

Modules

streamasync

Asynchronous analogs to the base *Chunker types that wrap Tokio’s AsyncRead types and implement Stream.

Structs

ByteChunker

The ByteChunker takes a bytes::Regex, wraps a byte source (that is, a type that implements std::io::Read) and iterates over chunks of bytes from that source that are delimited by the regular expression. It operates very much like bytes::Regex::split, except that it works on an incoming stream of bytes instead of a necessarily-already-in-memory slice.

CustomChunker

A chunker that has additionally been supplied with an Adapter, so it can produce arbitrary types. The CustomChunkers does not have a separate constructor; it is built by combining a ByteChunker with an Adapter using ByteChunker::with_adapter.

SimpleCustomChunker

A version of CustomChunker that takes a SimpleAdapter type.

StringAdapter

An example Adapter type for producing a chunker that yields Strings.

Enums

ErrorResponse

Type for specifying a Chunker’s behavior upon encountering an error.

MatchDisposition

Specify what the chunker should do with the matched text.

RcErr

Wraps various types of errors that can happen in the internals of a Chunker. The way Chunkers respond to and report these errors can be controlled through builder-pattern methods that take the ErrorResponse and Utf8FailureMode types.

Utf8FailureMode

Type for specifying a StringAdapter’s behavior upon encountering non-UTF-8 data.

Traits

Adapter

Trait used to implement a CustomChunker by transforming the output of a ByteChunker.

SimpleAdapter

Simpler, less flexible, version of the Adapter trait.