xstream-util-1.1.0 is not a library.
Visit the last successful build:
xstream-util-2.0.0
xstream
A command line tool to split a stream by a delimiter and pipe each section to a child process.
Installation
cargo install xstream-util
Benchmarks
For a simple illustration of the speed up for reasonably sized streams, the following simple benchmark compares generating 1001 streams of integers and summing them with bc
.
First, generate a null delimited set of streams with
; do ; ; done
This stream is roughly 50M, making each stream roughly 50k.
I then piped this into xstream
as
|
and xargs
as
|
which on my system gives:
Program | User | System | Elapsed |
---|---|---|---|
xstream |
6.55 | 1.53 | 0:06.91 |
xargs |
17.26 | 3.98 | 0:18.01 |
This benchmark is a toy example, but xstream
already provides a 60% speed up when each stream is only 50k.
To Do
- This currently only support single byte delimiters, but it shouldn't be too difficult to support multi-byte delimiters.
After storing the delimiter, finding it should work with
buf.windows().position(|&s| s == delim
. The tricky part will be not writing the lastdelim.len()
bytes in case the first part of delim is in them, but making sure that we do flush them if we've finished a stream. - Move the stream splitting into lib so that it can be tested. This will be especially crucial with multi-byte delimiters.