1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
/*!
This crate builds on the interfaces from [`yap`](https://crates.io/crates/yap) to allow simple parsing of streams.
# Why
There already exist [many](https://github.com/rosetta-rs/parse-rosetta-rs) crates that intend to help with parsing.
Of that list `nom`, `winnow`, `chumsky`, `combine` support parsing streams of values.
`nom`:
- No obvious way to signal the end of a stream to a parser.
- The user of the library has to implement a streaming parser noticeably differently from a non-streaming parser.
- Parsing occurs on chunks. Parsing dynamically sized chunks can require re-parsing the chunk from scratch and redoing work.
`winnow`:
- Parsing occurs on chunks. Parsing dynamically sized chunks can require re-parsing the chunk from scratch and redoing work.
`chumsky` is not designed for speed.
`combine` is complicated.
This crate allows using an already written [`yap`](https://crates.io/crates/yap) parser by simply changing the initial tokens declaration.
```rust
# #[cfg(feature = "alloc")] {
use std::{
fs::File,
io::{self, BufReader, Read},
};
use yap_streaming::{
// Allows you to use `.into_tokens()` on strings and slices,
// to get an instance of the above:
IntoTokens,
// Allows you to get an instance of `Tokens` that supports streams:
StrStreamTokens,
// This trait has all of the parsing methods on it:
Tokens,
};
// Write parser
// =========================================
#[derive(PartialEq, Debug)]
enum Op {
Plus,
Minus,
Multiply,
}
#[derive(PartialEq, Debug)]
enum OpOrDigit {
Op(Op),
Digit(u32),
}
// The `Tokens` trait builds on `Iterator`, so we get a `next` method.
fn parse_op(t: &mut impl Tokens<Item = char>) -> Option<Op> {
let loc = t.location();
match t.next()? {
'-' => Some(Op::Minus),
'+' => Some(Op::Plus),
'x' => Some(Op::Multiply),
_ => {
t.set_location(loc);
None
}
}
}
// We also get other useful functions..
fn parse_digits(t: &mut impl Tokens<Item = char>) -> Option<u32> {
t.take_while(|c| c.is_digit(10)).parse::<u32, String>().ok()
}
fn parse_all(t: &mut impl Tokens<Item = char>) -> impl Tokens<Item = OpOrDigit> + '_ {
// As well as combinator functions like `sep_by_all` and `surrounded_by`..
t.sep_by_all(
|t| {
t.surrounded_by(
|t| parse_digits(t).map(OpOrDigit::Digit),
|t| {
t.skip_while(|c| c.is_ascii_whitespace());
},
)
},
|t| parse_op(t).map(OpOrDigit::Op),
)
}
// Now we've parsed our input into OpOrDigits, let's calculate the result..
fn eval(t: &mut impl Tokens<Item = char>) -> u32 {
let op_or_digit = parse_all(t);
let mut current_op = Op::Plus;
let mut current_digit = 0;
for d in op_or_digit.into_iter() {
match d {
OpOrDigit::Op(op) => current_op = op,
OpOrDigit::Digit(n) => match current_op {
Op::Plus => current_digit += n,
Op::Minus => current_digit -= n,
Op::Multiply => current_digit *= n,
},
}
}
current_digit
}
// Use parser
// =========================================
// Get our input and convert into something implementing `Tokens`
let mut tokens = "10 + 2 x 12-4,foobar".into_tokens();
// Parse
assert_eq!(eval(&mut tokens), 140);
// Instead of parsing an in-memory buffer we can use `yap_streaming` to parse a stream.
// While we could [`std::io::Read::read_to_end()`] here, what if the file was too large
// to fit in memory? What if we were parsing from a network socket?
let mut io_err = None;
let file_chars = BufReader::new(File::open("examples/opOrDigit.txt").expect("open file"))
.bytes()
.map_while(|x| {
match x {
Ok(x) => {
if x.is_ascii() {
Some(x as char)
} else {
io_err = Some(io::ErrorKind::InvalidData.into());
// Don't parse any further if non-ascii input.
// This simple example parser only makes sense with ascii values.
None
}
}
Err(e) => {
io_err = Some(e);
// Don't parse any further if io error.
// Alternatively could panic, retry the byte,
// or include as an error variant and parse Result<char, ParseError> instead.
None
}
}
});
// Convert to something implementing `Tokens`.
// If parsing a stream not of `char` use [`yap_streaming::StreamTokens`] instead.
let mut tokens = StrStreamTokens::new(file_chars);
// Parse
assert_eq!(eval(&mut tokens), 140);
// Check that parse encountered no io errors.
assert!(io_err.is_none());
# }
```
*/
extern crate alloc;
pub use ;
pub use ;