oneyaml_syntax/parser/mod.rs
1//! The YAML 1.2.2 parser: source text to the lossless event stream.
2//!
3//! # Architecture
4//!
5//! * `scan` — the lexical layer: character classes (§5) and single-line
6//! scanners for indicators, properties, and plain scalar lines.
7//! * `scalar` — multi-line flow scalars (plain, single- and double-quoted)
8//! with folding and escapes.
9//! * `stream` — stream and document level: BOMs, directives, document
10//! markers (§9).
11//! * `block`, `head` — block collections and node heads (§8.2, §6.9).
12//! * `block_scalar` — literal and folded block scalars (§8.1).
13//! * `flow`, `flow_node`, `flow_block` — flow collections (§7.4), parsed
14//! by a sub-machine with its own explicit stack into a local buffer so a
15//! single-line flow collection can still become a block implicit key.
16//!
17//! The parser is an **explicit-stack state machine**: `State` says
18//! what the machine expects next and `Frame`s record the open documents and
19//! block collections. Nothing recurses, so arbitrarily deep input cannot
20//! overflow the call stack.
21//!
22//! **Placement (event contract C4).** Trivia (white space, comments, line
23//! breaks) is scanned into a *pending* buffer. Zero-width closing events
24//! (collection ends, implicit document ends, empty nodes) are pushed before
25//! the buffer is flushed, so they "end early"; zero-width openings are pushed
26//! after the flush, immediately before their first child, so they "start
27//! late".
28//!
29//! **Errors (C6).** A syntax error emits an [`EventKind::Error`] covering the
30//! rest of the offending line, completes the open structure with zero-width
31//! events (R2), and then skips whole lines as [`TriviaKind::Skipped`] until
32//! the next document marker, where parsing resumes (R4).
33//!
34//! # Streaming and buffering
35//!
36//! [`events_with`] is the parser; [`parse_with`] only collects it.
37//! What the parser holds between two pulls, for a consumer that drops
38//! each event (the bound is derived here; `tests/streaming.rs` checks it
39//! with `Events::retained_capacity`, hidden test instrumentation):
40//!
41//! * the flow hold (`flow.rs`): one line and at most the implicit-key
42//! window of productions 154-155, 1024 characters or `W` = 4096 bytes,
43//! whatever the collection's size. A byte yields at most 3 held items:
44//! an entry of `k` bytes plus its `,` yields at most `k + 1` items that
45//! cover bytes and 4 zero-width ones (a flow pair's start and end, an
46//! empty key and value), and `(k + 5) / (k + 1) <= 3`, reached by
47//! `[:,:,…]` and `[?,?,…]`. So the hold holds at most `3W` = 12,288
48//! items, and its open pair reservations at most one per entry, `W / 2`;
49//! * the output queue: a batch of 64 items plus the zero-width ends one
50//! token confirms (bounded by depth), **or**, once, the whole flow hold
51//! when the root is decided (or an error ends it) and the hold drains
52//! into it: at most `3W` items plus the batch and the depth-bounded
53//! ends. Both buffers keep their capacity, so for a single-line flow
54//! collection longer than the window both reach 12,288 items, which
55//! `std`'s power-of-two `VecDeque` rounds to a capacity of 16,384 each;
56//! * frame stacks, `deferred` ends and empty nodes, `flow_repair`, and the
57//! head buffer: bounded by depth;
58//! * trivia runs cost one queue item each and are lexed as they are
59//! pulled, including the white lines before a block scalar error;
60//! scalar values are decoded only by `Sourced::value`;
61//! * **not** bounded by depth or the key window: the `%TAG` handles of the
62//! current and the next document (one entry per directive).
63//!
64//! So, `%TAG` handles aside, the parser holds `O(depth)` plus at most
65//! `2 × 3W + W / 2` items, independent of the input's length; with
66//! amortised growth (capacity below twice the peak length) that is below
67//! `4 × 3W + W` = 53,248 slots plus `O(depth)`. Measured peak: ≈32,784.
68//!
69//! # Prefix timing conventions (not contract requirements)
70//!
71//! The event contract fixes *where* each event goes (C1-C6), not *when*,
72//! relative to a later error, the parser commits to it. yaml-test-suite
73//! records, for every invalid input, exactly the events its reference
74//! parser emitted before failing, and our harness compares that prefix
75//! byte for byte. The rules below reproduce that timing. They are
76//! conventions for matching the suite, justified by R5 ("the events before
77//! the first `Error` are what the parser was sure of"); they are **not**
78//! part of the contract, `check_contract` does not test them, and another
79//! choice would be equally valid structurally. Code comments cite them as
80//! T1-T5.
81//!
82//! * **T1: deferred implicit document start.** An implicit document's
83//! `DocumentStart` is emitted when its root node's content token is
84//! recognised, not when its properties are, so an error in the first
85//! head (e.g. `&a - b`) yields no document (`doc_start_deferred`).
86//! * **T2: deferred ends and empty nodes.** A block collection end caused
87//! by a strict dedent, and an empty node with properties, are emitted at
88//! once. A sequence end caused by a same-column non-`-` line, and an
89//! empty node without properties, are deferred until the next token is
90//! accepted; if it is not, they follow the `Error` as its repair
91//! (`deferred`).
92//! * **T3: key errors.** A node that turns out to be a multi-line implicit
93//! key never gets its completing event: a multi-line scalar key is
94//! rejected before it is emitted, and a multi-line flow collection key
95//! is rejected at its closing bracket (its contents are emitted, its end
96//! is not). A complete single-line quoted scalar or flow collection
97//! followed by a `:` that cannot start a mapping is emitted first, and
98//! the `:` is reported after it. A multi-line flow collection in a block
99//! key slot that no `:` follows (`'k' : ---\n[\n 'x' : value]`) is
100//! emitted whole; the missing `:` (`Y0018`) is reported where the scan
101//! stopped, which may be lines later or the end of input, so its
102//! diagnostic carries a secondary label covering the key.
103//! * **T4: the document stays open at content after the root.** A document
104//! ends only at its end marker, at the next document start, or at the end
105//! of the stream, so content after a complete root (whatever its style) is
106//! reported before the implicit `DocumentEnd`, which comes with the repair.
107//! * **T5: flow prefixes.** A flow collection with an error commits every
108//! event before the error, then the repairs.
109//!
110//! # Settings and limits
111//!
112//! [`parse_with`] takes the [`Settings`]. The parser enforces four of the
113//! [`Limits`](oneyaml_core::Limits); each is checked **before** the
114//! offending event is emitted, so the stream stays contract-valid and the
115//! error is repaired like any other (C6):
116//!
117//! * `max_depth` (`Y2000`): block and flow collections counted together.
118//! Checked when a collection opens, where a block collection that may
119//! not start there would be reported; for a flow collection that turns
120//! out to be an implicit key (whose depth grows by the mapping or flow
121//! pair it starts), when the key's `:` is seen.
122//! * `max_scalar_len` (`Y2001`): a scalar's value in bytes, checked before
123//! the scalar is emitted.
124//! * `max_document_size` (`Y2002`): the input length in bytes, checked
125//! first; the whole input is then skipped.
126//! * `max_documents` (`Y2003`): checked when a document would start; the
127//! rest of the input is then skipped.
128//!
129//! The help text of each limit diagnostic names the setting and its current
130//! value. The one leniency, `lenient_inline_colons`, is
131//! described in `head.rs`.
132
133mod block;
134mod block_scalar;
135mod flow;
136mod flow_block;
137mod flow_node;
138mod head;
139mod polish;
140mod read;
141mod run;
142mod scalar;
143mod scan;
144mod stream;
145
146/// How far past an implicit key's start the parser looks for its `:`
147/// (bounds how far a validation window reads past a peeked
148/// element's start).
149pub(crate) use flow::KEY_WINDOW;
150
151use std::collections::{BTreeMap, VecDeque};
152
153use oneyaml_core::code::{
154 DEPTH_LIMIT_EXCEEDED, DOCUMENT_TOO_LARGE, INTERNAL_ERROR, INVALID_COMMENT_CHARACTER,
155 INVALID_ESCAPE, MISPLACED_BYTE_ORDER_MARK, MISSING_MAPPING_COLON, SCALAR_TOO_LONG,
156 TAB_INDENTATION, TOO_MANY_DOCUMENTS, UNTERMINATED_FLOW_COLLECTION, UNTERMINATED_QUOTED_SCALAR,
157};
158use oneyaml_core::{Code, Diagnostic, Label, Settings, Span};
159
160use crate::event::{Event, EventKind, EventList, NodeInfo, Properties, Sourced, TriviaKind};
161
162pub(crate) use read::{
163 anchor_at, decoded_suffix, directive_at, directive_prefix, resolve_tag, scalar_value,
164 tag_source_at,
165};
166
167use scan::{break_len, byte, line_end, skip_white};
168
169/// Parses `source` into the lossless event stream with the default
170/// [`Settings`] (including [`Limits::default()`](oneyaml_core::Limits)).
171///
172/// Collects [`events`] into an [`EventList`]: the same parser, the same
173/// events. Never fails and never panics: problems are reported in-band as
174/// [`EventKind::Error`] and [`EventKind::Warning`] events, and the stream
175/// always satisfies the event contract (C1–C6). Runs in O(n) time with no
176/// recursion.
177#[must_use]
178pub fn parse(source: &str) -> EventList<'_> {
179 parse_with(source, &Settings::new())
180}
181
182/// Parses `source` into the lossless event stream under `settings`,
183/// collected into an [`EventList`] that keeps the source for reading (see
184/// [`events_with`]).
185///
186/// The parser reads the limits `max_depth`, `max_scalar_len`,
187/// `max_document_size`, and `max_documents`, the `lenient_inline_colons`
188/// leniency, and `honor_version_directive` (for the wording of the `%YAML
189/// 1.0` and `%YAML 1.1` warnings); every other setting belongs to a later layer. Same
190/// guarantees as [`parse`].
191#[must_use]
192pub fn parse_with<'a>(source: &'a str, settings: &Settings) -> EventList<'a> {
193 let events = events_with(source, settings)
194 .map(Sourced::into_event)
195 .collect();
196 EventList::new(source, events)
197}
198
199/// The lossless event stream of `source` under the default [`Settings`],
200/// produced incrementally (see [`Events`]).
201#[must_use]
202pub fn events(source: &str) -> Events<'_> {
203 events_with(source, &Settings::new())
204}
205
206/// The lossless event stream of `source` under `settings`, produced
207/// incrementally (see [`Events`]). The settings are read once; the
208/// iterator does not borrow them.
209#[must_use]
210pub fn events_with<'a>(source: &'a str, settings: &Settings) -> Events<'a> {
211 Events {
212 parser: Parser::new(source, settings),
213 }
214}
215
216/// A pull parser: an iterator over the lossless event stream of one source.
217///
218/// Each call to `next` runs the state machine only until the next event is
219/// known, so a consumer that drops events as it reads them (the serde
220/// layer) holds memory bounded by the nesting depth plus the grammar's
221/// lookahead, independent of the input's length. See the module
222/// docs for the bounds.
223///
224/// The stream is the one [`parse`] collects, and satisfies the event
225/// contract (C1–C6). Each event comes paired with the source as a
226/// [`Sourced`], so names and values are read from the right text by
227/// construction. The iterator is fused: after `StreamEnd` it returns
228/// `None` forever.
229#[derive(Debug)]
230pub struct Events<'a> {
231 parser: Parser<'a>,
232}
233
234impl<'a> Iterator for Events<'a> {
235 type Item = Sourced<'a>;
236
237 fn next(&mut self) -> Option<Sourced<'a>> {
238 let p = &mut self.parser;
239 if let Some(event) = p.out.pop(p.src) {
240 return Some(event);
241 }
242 p.fill();
243 p.out.pop(p.src)
244 }
245}
246
247impl core::iter::FusedIterator for Events<'_> {}
248
249impl<'a> Events<'a> {
250 /// The source text being parsed.
251 #[must_use]
252 pub const fn source(&self) -> &'a str {
253 self.parser.src
254 }
255
256 /// Test instrumentation, not part of the API: the total size, in
257 /// elements, of what the parser holds right now: the capacity of the
258 /// output queue, the flow-collection hold, and the stacks and owed
259 /// events that grow with nesting depth, plus the number of `%TAG`
260 /// handles held for the current and the next document.
261 ///
262 /// For a consumer that drops each event, everything but the `%TAG`
263 /// handles is bounded by the nesting depth plus the implicit-key
264 /// lookahead (see *Streaming and buffering* in the module docs), never
265 /// by the input's length. The handles grow with the number of `%TAG`
266 /// directives of one document, which the input chooses; they are
267 /// counted so that tests see them, and the bound is stated without
268 /// them. Hidden from the docs: `tests/streaming.rs` checks the bound
269 /// through it.
270 #[doc(hidden)]
271 #[must_use]
272 pub fn retained_capacity(&self) -> usize {
273 let p = &self.parser;
274 p.out
275 .q
276 .capacity()
277 .saturating_add(p.deferred.capacity())
278 .saturating_add(p.stack.capacity())
279 .saturating_add(p.flow_repair.capacity())
280 .saturating_add(p.scratch.capacity())
281 .saturating_add(p.flow.capacity())
282 .saturating_add(p.tags.len())
283 .saturating_add(p.pending_tags.len())
284 }
285}
286
287/// How many items the parser queues ahead before handing events out: a
288/// constant, so the queue stays bounded, and large enough that the
289/// state machine runs in batches rather than once per event.
290const BATCH: usize = 64;
291
292/// An entry of the output queue: an event, or a run of trivia (white
293/// space, line breaks, comments, byte order marks) that is lexed into
294/// events only as the consumer pulls them, so any amount of trivia costs
295/// O(1) memory.
296#[derive(Clone, Debug)]
297pub(super) enum Item<'a> {
298 Event(Event<'a>),
299 /// Trivia covering `start..end`.
300 Trivia(usize, usize),
301}
302
303/// The output queue, in stream order.
304#[derive(Debug, Default)]
305pub(super) struct Out<'a> {
306 q: VecDeque<Item<'a>>,
307 /// What is left of the trivia run being lexed (empty when `.0 >= .1`);
308 /// it comes before everything in `q`. Held here, not pushed back on the
309 /// queue, so a run of several pieces (a line break and the next line's
310 /// indentation, a comment) costs no queue traffic between them.
311 run: (usize, usize),
312}
313
314impl<'a> Out<'a> {
315 pub(super) fn event(&mut self, event: Event<'a>) {
316 self.q.push_back(Item::Event(event));
317 }
318
319 /// Queues the trivia covering `start..end` (nothing when empty).
320 pub(super) fn trivia(&mut self, start: usize, end: usize) {
321 if start < end {
322 self.q.push_back(Item::Trivia(start, end));
323 }
324 }
325
326 pub(super) fn item(&mut self, item: Item<'a>) {
327 match item {
328 Item::Event(event) => self.event(event),
329 Item::Trivia(start, end) => self.trivia(start, end),
330 }
331 }
332
333 /// Enough is queued to hand out (see [`BATCH`]).
334 fn is_full(&self) -> bool {
335 self.q.len() >= BATCH
336 }
337
338 /// The next event, lexing the front trivia run if needed.
339 #[inline]
340 fn pop(&mut self, src: &'a str) -> Option<Sourced<'a>> {
341 // The `Sourced` is built at each return site, so the item is
342 // written once, straight into the caller's slot.
343 let (from, to) = if self.run.0 < self.run.1 {
344 self.run
345 } else {
346 match self.q.pop_front()? {
347 Item::Event(event) => return Some(Sourced::new(event, src)),
348 Item::Trivia(from, to) => (from, to),
349 }
350 };
351 match scan::lex_trivia(src, from, to) {
352 Some((kind, stop)) => {
353 self.run = (stop, to);
354 let span = Span::new(from, stop);
355 Some(Sourced::new(Event::new(EventKind::Trivia(kind), span), src))
356 }
357 None => {
358 self.run = (to, to);
359 Some(Sourced::new(broken_trivia_run(from, to), src))
360 }
361 }
362 }
363}
364
365/// Only trivia is ever queued as a run. In release the rest of the
366/// run becomes an in-band Y9000 error, never a silently mislabelled trivia
367/// event.
368#[cold]
369#[inline(never)]
370fn broken_trivia_run<'a>(from: usize, to: usize) -> Event<'a> {
371 debug_assert!(false, "non-trivia byte in a trivia run at {from}");
372 let span = Span::new(from, to);
373 Event::new(
374 EventKind::Error(internal("non-trivia bytes in a trivia run", span)),
375 span,
376 )
377}
378
379/// Where a block mapping is within its current entry.
380#[derive(Clone, Copy, Debug, PartialEq, Eq)]
381enum MapPhase {
382 /// Expecting a key or the end of the mapping: `ns-l-block-map-entry`
383 /// (production 188) via `?` (190), `:` after an empty key, or an
384 /// implicit key (192-194).
385 Key,
386 /// The key after `?` is being parsed (`c-l-block-map-explicit-key`,
387 /// production 190).
388 ExplicitKey,
389 /// An explicit key is complete; expecting `l-block-map-explicit-value`
390 /// (production 191) or the next entry.
391 AfterExplicitKey,
392 /// The value after `:` is being parsed (productions 191 and 194).
393 Value,
394}
395
396/// An open document or block collection.
397#[derive(Clone, Copy, Debug)]
398enum Frame {
399 /// `l-any-document` (production 210); the root node is complete.
400 Document { root_done: bool },
401 /// `l+block-sequence(n)` (production 183) with entries at `indent`.
402 Seq { indent: usize },
403 /// `l+block-mapping(n)` (production 187) with keys at `indent`.
404 Map { indent: usize, phase: MapPhase },
405}
406
407/// Context for parsing one node (`s-l+block-node(n,c)`, production 196).
408#[derive(Clone, Copy, Debug)]
409struct NodeCtx {
410 /// Indentation `n` of the parent collection; `None` is the document
411 /// level (indentation -1).
412 parent: Option<usize>,
413 /// A compact block collection may start on the current line
414 /// (`s-l+block-indented`, production 185: after `-`, `?`, or an explicit
415 /// `:`).
416 compact: bool,
417 /// A block sequence may sit at the parent's own indentation
418 /// (`seq-space(n, block-out)`, production 201).
419 seq_at_parent: bool,
420}
421
422/// What the machine does next.
423#[derive(Clone, Copy, Debug)]
424enum State {
425 /// Between documents (`l-yaml-stream`, production 211).
426 Stream,
427 /// Expecting a node in the context given.
428 Node(NodeCtx),
429 /// A node or indicator is done; the innermost frame decides: another
430 /// entry (productions 183-184, 187-194), the collection's end, or the
431 /// document's end (`l-document-suffix`, production 205).
432 Next,
433 /// Inside a flow collection opened by a block node head (§7.4); the
434 /// flow sub-machine (`flow.rs`) holds its own explicit stack.
435 Flow,
436 /// Resynchronising after an error (C6 R4): whole lines are skipped up
437 /// to the next `c-directives-end`/`c-document-end` (203-204).
438 Skip,
439 /// `StreamEnd` emitted.
440 Done,
441}
442
443#[derive(Debug)]
444struct Parser<'a> {
445 src: &'a str,
446 /// Events ready for the consumer (bounded by one step's output).
447 out: Out<'a>,
448 /// End of the last emitted event.
449 emit_pos: usize,
450 /// Zero-width events owed at `emit_pos` (block collection ends and
451 /// empty nodes), emitted lazily: before the pending trivia once the next
452 /// token is accepted, or after the `Error` if it is not. Timing
453 /// convention T2 (module docs), not a contract rule. Bounded by depth.
454 deferred: Vec<EventKind>,
455 /// Scan position. The bytes `emit_pos..pos` are trivia scanned but not
456 /// yet emitted ("pending"); they are queued as one lazily lexed run.
457 pos: usize,
458 /// Start of the current line.
459 line_start: usize,
460 /// Leading spaces of the current line.
461 line_indent: usize,
462 /// No content has been consumed on the current line yet.
463 fresh: bool,
464 /// A tab occurred in white space since the last line break (or since the
465 /// last consumed token on this line).
466 tab_seen: bool,
467 stack: Vec<Frame>,
468 state: State,
469 /// Properties emitted as trivia but not yet attached to a node.
470 props: Properties,
471 /// An implicit document was entered but its `DocumentStart` is deferred
472 /// until its first node event (so an error in the first node's head
473 /// yields no document at all). Timing convention T1 (module docs).
474 doc_start_deferred: bool,
475 /// Directives seen since the last document.
476 directives_pending: bool,
477 /// A `%YAML` directive was seen for the next document.
478 yaml_seen: bool,
479 /// `%TAG` handles declared for the next document, with their prefixes
480 /// (read once).
481 pending_tags: BTreeMap<&'a str, scan::TagPrefix>,
482 /// `%TAG` handles of the current document.
483 tags: BTreeMap<&'a str, scan::TagPrefix>,
484 /// A `...` was just consumed; the rest of its line must be trivia.
485 doc_end_line: bool,
486 /// A document was ended by a byte order mark (`prefix_mark_at`), not
487 /// by `...`: until the next `---`, only a document prefix may follow
488 /// (production 211's second branch).
489 /// Where that mark starts (for the diagnostic).
490 prefix_without_end: Option<usize>,
491 /// A byte order mark that caused the error about to be reported
492 /// (`mark_then_fail`): its `MISPLACED_BYTE_ORDER_MARK` is reported with
493 /// the error, in the order of their labels. Consumed
494 /// by `fail_with`.
495 mark_before_error: Option<usize>,
496 /// Where the first character that cannot belong to any valid document
497 /// is, when that is later than the scan position the failing construct
498 /// started at: the error is reported there (consumed by `fail_with`).
499 /// A cell, so that scanners holding only `&self` can name it.
500 blame: core::cell::Cell<Option<usize>>,
501 /// Zero-width events completing an erroneous flow collection, emitted
502 /// right after its `Error` (C6 R2).
503 flow_repair: Vec<Owed>,
504 /// Reused buffer for a node head's events (no allocation per
505 /// node on the hot path). At most a few events: one line's properties.
506 scratch: Vec<Event<'a>>,
507 /// The open flow collection, if any (`State::Flow`).
508 flow: flow::FlowState<'a>,
509 /// `Limits::max_depth`.
510 max_depth: usize,
511 /// `Limits::max_scalar_len`.
512 max_scalar_len: usize,
513 /// `Limits::max_document_size`.
514 max_document_size: usize,
515 /// `Limits::max_documents`.
516 max_documents: usize,
517 /// `Settings::lenient_inline_colons`.
518 lenient_inline_colons: bool,
519 /// `Settings::honor_version_directive`, for the `%YAML 1.0`
520 /// and `%YAML 1.1` warnings' wording only.
521 honor_version_directive: bool,
522 /// Documents started so far.
523 documents: usize,
524 /// A stream-wide limit was exceeded: skip to the end of the input, not
525 /// just to the next document marker.
526 halted: bool,
527 /// Start and end of the last quoted scalar emitted as a block node, so
528 /// content glued to its closing quote (`'it's'`) gets a targeted
529 /// diagnostic.
530 last_quoted: (usize, usize),
531}
532
533/// A zero-width event owed after an error (C6 R2), packed once its position
534/// is known.
535#[derive(Clone, Debug)]
536pub(super) enum Owed {
537 /// A collection end.
538 Kind(EventKind),
539 /// An empty plain scalar with these properties.
540 Empty(Properties),
541}
542
543/// `kind` with its node information set (for a node start or scalar).
544pub(super) fn with_info(mut kind: EventKind, new: NodeInfo) -> EventKind {
545 if let EventKind::Scalar { info, .. }
546 | EventKind::MappingStart { info, .. }
547 | EventKind::SequenceStart { info, .. } = &mut kind
548 {
549 *info = new;
550 }
551 kind
552}
553
554/// The `Y9000` diagnostic for a broken internal invariant: the release-build
555/// outcome of a `debug_assert!`.
556pub(super) fn internal(what: &str, span: Span) -> Box<Diagnostic> {
557 Box::new(
558 Diagnostic::error(
559 INTERNAL_ERROR,
560 format!("internal error: {what}"),
561 Label::new(span, ""),
562 )
563 .with_help("this is a bug in oneyaml; please report it with this input"),
564 )
565}
566
567/// The help every diagnostic with `code` gets unless its producer
568/// refines it (the diagnostic polish list: what to write instead).
569fn default_help(code: Code) -> Option<&'static str> {
570 Some(match code {
571 TAB_INDENTATION => {
572 "YAML indents with spaces only; replace the tabs used for indentation here with spaces"
573 }
574 MISSING_MAPPING_COLON => {
575 "each entry of a block mapping is written `key: value`, with a space after the `:`"
576 }
577 UNTERMINATED_QUOTED_SCALAR => {
578 "add the closing quote; inside '...' write '' for a literal ', inside \"...\" write \\\" for a literal \""
579 }
580 INVALID_ESCAPE => {
581 "in double quotes, `\\` starts an escape such as `\\n`, `\\t`, `\\\"` or `\\\\`; write `\\\\` for a literal backslash, or use single quotes, where `\\` has no special meaning"
582 }
583 UNTERMINATED_FLOW_COLLECTION => "close every `[` with `]` and every `{` with `}`",
584 _ => return None,
585 })
586}
587
588/// The `MISPLACED_BYTE_ORDER_MARK` diagnostic for the mark at `start..end`
589/// (§5.2, production 202).
590pub(super) fn misplaced_bom_diagnostic(start: usize, end: usize) -> Box<Diagnostic> {
591 Box::new(
592 Diagnostic::error(
593 MISPLACED_BYTE_ORDER_MARK,
594 "a byte order mark may appear only at the start of a document prefix",
595 Label::new(Span::new(start, end), ""),
596 )
597 .with_help(
598 "remove this byte order mark (U+FEFF); a mark is allowed only at the very start of the stream, or at the start of a line before a document",
599 ),
600 )
601}
602
603/// Whether `bytes` may hold a character that is not printable: false only
604/// for bytes that are all ASCII, none a control character (a tab counts as
605/// one here) or DEL. Eight bytes at a time.
606#[inline(never)]
607fn may_hold_unprintable(bytes: &[u8]) -> bool {
608 const ONES: u64 = 0x0101_0101_0101_0101;
609 const HIGH: u64 = 0x8080_8080_8080_8080;
610 let mut words = bytes.chunks_exact(8);
611 for chunk in words.by_ref() {
612 let Ok(chunk) = <[u8; 8]>::try_from(chunk) else {
613 return true;
614 };
615 let w = u64::from_le_bytes(chunk);
616 // A byte with the high bit set, below 0x20, or equal to 0x7f.
617 let del = w ^ (ONES * 0x7f);
618 let found = w | (w.wrapping_sub(ONES * 0x20) & !w) | (del.wrapping_sub(ONES) & !del);
619 if found & HIGH != 0 {
620 return true;
621 }
622 }
623 words.remainder().iter().any(|b| *b < 0x20 || *b >= 0x7f)
624}
625
626/// The error for the first character of the comment `start..end` that is
627/// not printable, if any.
628#[inline(never)]
629pub(super) fn comment_char_diagnostic(
630 src: &str,
631 start: usize,
632 end: usize,
633) -> Option<Box<Diagnostic>> {
634 let text = src.get(start..end)?;
635 // Nearly every comment is plain printable ASCII: a cheap pass over the
636 // bytes rules that out before the slower pass over characters.
637 if !may_hold_unprintable(text.as_bytes()) {
638 return None;
639 }
640 let (k, c) = text.char_indices().find(|(_, c)| !scan::is_printable(*c))?;
641 let at = start.saturating_add(k);
642 Some(Box::new(
643 Diagnostic::error(
644 INVALID_COMMENT_CHARACTER,
645 "a comment may hold only printable characters",
646 Label::new(Span::new(at, at.saturating_add(c.len_utf8())), ""),
647 )
648 .with_help("remove this character; control characters (U+0000 to U+001F except tab, U+007F, and U+0080 to U+009F except U+0085) cannot appear in a comment"),
649 ))
650}
651
652fn diagnostic(code: Code, message: &str, span: Span) -> Box<Diagnostic> {
653 Box::new(Diagnostic::error(code, message, Label::new(span, "")))
654}
655
656impl<'a> Parser<'a> {
657 fn new(src: &'a str, settings: &Settings) -> Self {
658 let limits = settings.get_limits();
659 let mut p = Self {
660 src,
661 out: Out::default(),
662 emit_pos: 0,
663 deferred: Vec::new(),
664 pos: 0,
665 line_start: 0,
666 line_indent: 0,
667 fresh: true,
668 tab_seen: false,
669 stack: Vec::new(),
670 state: State::Stream,
671 props: Properties::none(),
672 doc_start_deferred: false,
673 directives_pending: false,
674 yaml_seen: false,
675 pending_tags: BTreeMap::new(),
676 tags: BTreeMap::new(),
677 doc_end_line: false,
678 prefix_without_end: None,
679 mark_before_error: None,
680 blame: core::cell::Cell::new(None),
681 flow_repair: Vec::new(),
682 scratch: Vec::new(),
683 flow: flow::FlowState::default(),
684 max_depth: limits.get_max_depth(),
685 max_scalar_len: limits.get_max_scalar_len(),
686 max_document_size: limits.get_max_document_size(),
687 max_documents: limits.get_max_documents(),
688 lenient_inline_colons: settings.get_lenient_inline_colons(),
689 honor_version_directive: settings.get_honor_version_directive(),
690 documents: 0,
691 halted: false,
692 last_quoted: (0, 0),
693 };
694 p.push(EventKind::StreamStart, 0, 0);
695 if src.len() > p.max_document_size {
696 // Refuse oversized input before parsing any of it.
697 p.halt(DOCUMENT_TOO_LARGE, "the input is too large");
698 }
699 p
700 }
701
702 /// Runs the state machine until [`BATCH`] items are queued or the
703 /// stream is complete. Not inlined, so that `Events::next`, which
704 /// mostly just pops the queue, stays small (per-event overhead).
705 #[inline(never)]
706 fn fill(&mut self) {
707 // One loop over an explicit state; no recursion on input.
708 while !self.out.is_full() && !matches!(self.state, State::Done) {
709 self.step();
710 }
711 }
712
713 /// One transition of the state machine. Each queues a bounded number
714 /// of items: one token or line and the zero-width events it
715 /// confirms, which are bounded by the nesting depth.
716 ///
717 fn step(&mut self) {
718 match self.state {
719 State::Stream => self.stream(),
720 State::Node(ctx) => self.node(ctx),
721 State::Next => self.next(),
722 State::Flow => self.flow_step(),
723 State::Skip => self.skip(),
724 State::Done => {}
725 }
726 }
727
728 // ------------------------------------------------------------ emission
729
730 /// Emits an event covering `start..end` (`start` must be `emit_pos`).
731 fn push(&mut self, kind: EventKind, start: usize, end: usize) {
732 self.push_event(Event::new(kind, Span::new(start, end)));
733 }
734
735 /// Emits a built event (its start must be `emit_pos`).
736 fn push_event(&mut self, event: Event<'a>) {
737 self.emit_pos = event.span().end;
738 self.out.event(event);
739 }
740
741 /// Emits a zero-width event at `emit_pos` (before any pending trivia),
742 /// after any deferred ones.
743 fn zero(&mut self, kind: EventKind) {
744 self.materialize();
745 let at = self.emit_pos;
746 self.push(kind, at, at);
747 }
748
749 /// [`Parser::zero`] for a node event with properties (packed relative
750 /// to its position, `emit_pos`).
751 fn zero_node(&mut self, kind: EventKind, props: Properties) {
752 let info = NodeInfo::new(self.emit_pos, props, None);
753 self.zero(with_info(kind, info));
754 }
755
756 /// Owes a zero-width event at `emit_pos` (see `deferred`).
757 fn defer(&mut self, kind: EventKind) {
758 self.deferred.push(kind);
759 }
760
761 /// Emits the deferred zero-width events.
762 fn materialize(&mut self) {
763 if self.deferred.is_empty() {
764 return;
765 }
766 let at = self.emit_pos;
767 for kind in self.deferred.drain(..) {
768 self.out.event(Event::new(kind, Span::empty(at)));
769 }
770 }
771
772 /// Emits the deferred events, then the pending trivia.
773 fn flush(&mut self) {
774 self.materialize();
775 self.out.trivia(self.emit_pos, self.pos);
776 self.emit_pos = self.pos;
777 }
778
779 /// Flushes and, for a deferred implicit document, emits its start: call
780 /// before the first event of any node.
781 fn begin_content(&mut self) {
782 self.flush();
783 if self.doc_start_deferred {
784 self.doc_start_deferred = false;
785 self.zero(EventKind::document_start(false));
786 }
787 }
788
789 /// Emits an event at the scan position after flushing, and consumes it.
790 fn token(&mut self, kind: EventKind, end: usize) {
791 let start = self.pos;
792 self.push(kind, start, end);
793 self.pos = end;
794 self.fresh = false;
795 self.tab_seen = false;
796 }
797
798 /// The reusable head buffer, empty.
799 fn take_scratch(&mut self) -> Vec<Event<'a>> {
800 let mut events = core::mem::take(&mut self.scratch);
801 events.clear();
802 events
803 }
804
805 /// Commits a head buffer and keeps its capacity for the next head.
806 fn commit_scratch(&mut self, mut events: Vec<Event<'a>>) {
807 self.commit(events.drain(..));
808 self.scratch = events;
809 }
810
811 /// An empty plain scalar at `emit_pos` carrying the pending properties.
812 /// With properties the node is certain and is emitted at once; without,
813 /// it is deferred until the next token confirms it (timing convention
814 /// T2).
815 fn empty(&mut self) {
816 let props = core::mem::take(&mut self.props);
817 if props.is_empty() {
818 self.defer(EventKind::plain());
819 } else {
820 self.zero_node(EventKind::plain(), props);
821 }
822 }
823
824 fn col(&self) -> usize {
825 self.pos.saturating_sub(self.line_start)
826 }
827
828 // ------------------------------------------------------------ limits
829
830 /// The number of open block collections. The stack is a `Document`
831 /// frame followed by the block collections inside it.
832 fn block_depth(&self) -> usize {
833 self.stack.len().saturating_sub(1)
834 }
835
836 /// `Limits::max_depth` allows a collection at `depth`.
837 fn depth_ok(&self, depth: usize) -> bool {
838 depth <= self.max_depth
839 }
840
841 /// `Limits::max_scalar_len` allows a scalar value of `len` bytes.
842 fn scalar_len_ok(&self, len: usize) -> bool {
843 len <= self.max_scalar_len
844 }
845
846 /// The help of a limit diagnostic names the setting to raise and
847 /// its current value.
848 fn limit_help(&self, code: Code) -> Option<String> {
849 let (setting, value) = match code {
850 DEPTH_LIMIT_EXCEEDED => ("max_depth", self.max_depth),
851 SCALAR_TOO_LONG => ("max_scalar_len", self.max_scalar_len),
852 DOCUMENT_TOO_LARGE => ("max_document_size", self.max_document_size),
853 TOO_MANY_DOCUMENTS => ("max_documents", self.max_documents),
854 _ => return None,
855 };
856 Some(format!("raise Limits::{setting} (currently {value})"))
857 }
858
859 /// Reports a stream-wide limit and skips the rest of the input.
860 fn halt(&mut self, code: Code, message: &str) {
861 self.fail(code, message, false);
862 self.halted = true;
863 }
864
865 // ------------------------------------------------------------ trivia
866
867 /// `#` at `i` starts a comment (production 75).
868 fn comment_at(&self, i: usize) -> bool {
869 scan::comment_at(self.src, i)
870 }
871
872 /// Scans white space, comments, and line breaks as pending trivia
873 /// (`s-l-comments`, production 79, and `s-separate`, production 80).
874 ///
875 /// Byte order marks (§5.2, productions 202 and 211): `bom` says the
876 /// scan is between documents, where `l-document-prefix` may start.
877 /// There a mark at the very start of a line (nothing before it on the
878 /// line, no directives pending) is `ByteOrderMark` trivia and the
879 /// line's columns count after it. Inside a document such a mark
880 /// ([`Parser::prefix_mark_at`]) ends the document like `---` does
881 /// too: the scan stops before it, and the structure closes
882 /// around it. Anywhere else a mark is a recoverable
883 /// `MISPLACED_BYTE_ORDER_MARK` error, and the parse goes on
884 /// as if it were absent (`misplaced_bom`). A mark inside a comment
885 /// stays in the comment token and is reported after it
886 /// (`marks_in_token`).
887 fn scan_trivia(&mut self, bom: bool) {
888 loop {
889 let i = self.pos;
890 let kind_end = match byte(self.src, i) {
891 Some(b' ' | b'\t') => {
892 // Leading spaces of the run, then (rarely) tabs.
893 let spaces = self
894 .src
895 .as_bytes()
896 .get(i..)
897 .map_or(0, |r| r.iter().take_while(|b| **b == b' ').count());
898 let mut end = i.saturating_add(spaces);
899 if byte(self.src, end) == Some(b'\t') {
900 self.tab_seen = true;
901 end = skip_white(self.src, end);
902 }
903 // Leading spaces of the line: the run starts where the
904 // line's spaces so far end, which is the line start
905 // unless a skipped misplaced mark sits in between (its
906 // bytes were moved out of the line by `misplaced_bom`).
907 if i == self.line_start.saturating_add(self.line_indent) {
908 self.line_indent = self.line_indent.saturating_add(spaces);
909 }
910 (TriviaKind::Whitespace, end)
911 }
912 Some(b'\n' | b'\r') => {
913 let end = i.saturating_add(break_len(self.src, i));
914 self.line_start = end;
915 self.line_indent = 0;
916 self.fresh = true;
917 self.tab_seen = false;
918 (TriviaKind::LineBreak, end)
919 }
920 Some(b'#') if self.comment_at(i) => {
921 // c-nb-comment-text (production 75) is `nb-char`s,
922 // which exclude the mark (production 27).
923 let (end, clean) = scan::comment_extent(self.src, i);
924 self.pos = end;
925 // Printable ASCII holds no byte order mark and no
926 // character to report.
927 if !clean {
928 let bad = comment_char_diagnostic(self.src, i, end);
929 self.marks_in_token(i, end, bad);
930 }
931 continue;
932 }
933 _ if !bom && self.prefix_mark_at(i) => break,
934 _ if scan::mark_at(self.src, i) => {
935 let end = i.saturating_add('\u{feff}'.len_utf8());
936 // Any number of document prefixes may precede a document,
937 // and each may start with a mark (productions 211, 202),
938 // so a mark right behind another is a prefix start too.
939 let prefix_start = bom
940 && self.fresh
941 && i == self.line_start
942 && self.line_indent == 0
943 && !self.directives_pending;
944 if !prefix_start {
945 self.misplaced_bom(i, end);
946 continue;
947 }
948 // §5.2: the mark is no column; the line starts after it.
949 self.line_start = end;
950 self.line_indent = 0;
951 (TriviaKind::ByteOrderMark, end)
952 }
953 _ => break,
954 };
955 // Pending trivia is queued later as one lazy run (`flush`).
956 let (_, end) = kind_end;
957 self.pos = end;
958 }
959 }
960
961 /// A byte order mark at `i` at the very start of a fresh line inside a
962 /// document: it can continue nothing (it is no `ns-char`), so it ends
963 /// the document implicitly and starts an `l-document-prefix`
964 /// (production 211, §9.1.1). What may follow the prefix is
965 /// checked in `stream` (`prefix_without_end`).
966 pub(super) fn prefix_mark_at(&self, i: usize) -> bool {
967 self.fresh
968 && i == self.line_start
969 && self.line_indent == 0
970 && !self.stack.is_empty()
971 && scan::mark_at(self.src, i)
972 }
973
974 /// Reports each byte order mark inside the token `start..end` (a
975 /// comment, a block scalar), which keeps its bytes: one
976 /// zero-width `MISPLACED_BYTE_ORDER_MARK` error per mark right after
977 /// the token, whose label covers the mark. The pending
978 /// trivia up to the scan position is flushed first, so the error
979 /// directly follows the token and nothing lies between them.
980 pub(super) fn marks_in_token(
981 &mut self,
982 start: usize,
983 end: usize,
984 bad: Option<Box<Diagnostic>>,
985 ) {
986 let text = self.src.get(start..end).unwrap_or_default();
987 if bad.is_none() && !text.contains('\u{feff}') {
988 return;
989 }
990 self.flush();
991 // The errors keep the order of the characters they report.
992 let mut bad = bad;
993 for (k, _) in text.match_indices('\u{feff}') {
994 let at = start.saturating_add(k);
995 if bad.as_ref().is_some_and(|d| d.primary.span.start < at) {
996 self.comment_error(bad.take());
997 }
998 let diag = misplaced_bom_diagnostic(at, at.saturating_add('\u{feff}'.len_utf8()));
999 let here = self.emit_pos;
1000 self.push(EventKind::Error(diag), here, here);
1001 }
1002 self.comment_error(bad);
1003 }
1004
1005 /// Pushes the error for a character that cannot be in a comment, if
1006 /// there is one, right after the comment (production 1; comment text
1007 /// is `nb-char`s, productions 75 and 27): one zero-width
1008 /// `INVALID_COMMENT_CHARACTER` error whose label covers the character.
1009 /// The comment itself is kept as written, and the parse goes on.
1010 fn comment_error(&mut self, bad: Option<Box<Diagnostic>>) {
1011 if let Some(diag) = bad {
1012 let here = self.emit_pos;
1013 self.push(EventKind::Error(diag), here, here);
1014 }
1015 }
1016
1017 /// A byte order mark at `start..end` where no document prefix may
1018 /// start (§5.2, production 202): a recoverable
1019 /// `MISPLACED_BYTE_ORDER_MARK` error covering the mark, after the
1020 /// pending trivia (deferred zero-width events go first, at their own
1021 /// position). The parse then continues as if the mark were absent
1022 /// between tokens: the line start moves by the mark's length, so
1023 /// columns and the line's leading spaces are counted without it, and
1024 /// `fresh` is unchanged. (A mark right after a plain scalar's text ends
1025 /// that scalar, as any non-`ns-char` does; what follows is new
1026 /// content. Marks inside comments and block scalars: `marks_in_token`.)
1027 fn misplaced_bom(&mut self, start: usize, end: usize) {
1028 self.flush();
1029 self.push(
1030 EventKind::Error(misplaced_bom_diagnostic(start, end)),
1031 start,
1032 end,
1033 );
1034 self.pos = end;
1035 self.line_start = self
1036 .line_start
1037 .saturating_add(end.saturating_sub(start))
1038 .min(end);
1039 }
1040
1041 // ------------------------------------------------------------ structure
1042
1043 /// A child node of the innermost frame is complete.
1044 fn child_done(&mut self) {
1045 match self.stack.last_mut() {
1046 Some(Frame::Document { root_done }) => *root_done = true,
1047 Some(Frame::Map { phase, .. }) => {
1048 *phase = match *phase {
1049 MapPhase::ExplicitKey => MapPhase::AfterExplicitKey,
1050 MapPhase::AfterExplicitKey | MapPhase::Value | MapPhase::Key => MapPhase::Key,
1051 };
1052 }
1053 Some(Frame::Seq { .. }) | None => {}
1054 }
1055 self.state = State::Next;
1056 }
1057
1058 /// Closes the innermost block collection with a zero-width end. `certain`:
1059 /// the end is confirmed (a strict dedent or the end of the document) and
1060 /// is emitted at once; otherwise it is deferred until the next token is
1061 /// accepted by an outer collection (timing convention T2).
1062 fn close_block(&mut self, certain: bool) {
1063 let end = match self.stack.pop() {
1064 Some(Frame::Map { .. }) => EventKind::MappingEnd,
1065 Some(Frame::Seq { .. }) => EventKind::SequenceEnd,
1066 Some(frame @ Frame::Document { .. }) => {
1067 self.stack.push(frame);
1068 self.child_done();
1069 return;
1070 }
1071 None => {
1072 self.child_done();
1073 return;
1074 }
1075 };
1076 if certain {
1077 self.zero(end);
1078 } else {
1079 self.defer(end);
1080 }
1081 self.child_done();
1082 }
1083
1084 // ------------------------------------------------------------ errors
1085
1086 /// Reports a syntax error at the scan position covering the rest of the
1087 /// line, repairs the structure (C6 R2), and starts resynchronising (R4).
1088 /// `slot_open`: a node is owed to the innermost frame.
1089 fn fail(&mut self, code: Code, message: &str, slot_open: bool) {
1090 self.fail_with(code, message, slot_open, |d| d);
1091 }
1092
1093 /// [`Parser::fail`] with a diagnostic refined by `refine` (secondary
1094 /// labels, help, suggestions). Limit help is added automatically.
1095 fn fail_with(
1096 &mut self,
1097 code: Code,
1098 message: &str,
1099 slot_open: bool,
1100 refine: impl FnOnce(Diagnostic) -> Diagnostic,
1101 ) {
1102 let mark = self.mark_before_error.take();
1103 let blame = self.blame.take();
1104 if matches!(self.state, State::Skip) {
1105 // Already failed in this step: one error per problem, and never
1106 // a second error that could swallow the next line (e.g. `---`).
1107 // Emit anything scanned since, so the stream keeps tiling (C1)
1108 // when `skip` resumes at `pos`.
1109 self.flush();
1110 if self.emit_pos != self.pos || !self.deferred.is_empty() {
1111 // `flush` guarantees this; never a panic: a Y9000 event in
1112 // release.
1113 debug_assert!(false, "flush left owed events or pending trivia");
1114 self.internal_error("`flush` left owed events or pending trivia");
1115 }
1116 return;
1117 }
1118 // Deferred events were not confirmed by an accepted token: they go
1119 // after the error, as part of its repair (R2, R5).
1120 let deferred = core::mem::take(&mut self.deferred);
1121 self.flush();
1122 self.deferred = deferred;
1123 let start = self.pos;
1124 // Without a mark named by the failing construct, a mark
1125 // on the rest of the offending line that no quote precedes (so it
1126 // is not inside a quoted scalar, whose `nb-json` allows it) is
1127 // reported too. Limit and internal errors are about something
1128 // else.
1129 let mark = mark.or_else(|| {
1130 let about_syntax = code.get() < 1000;
1131 let text = self.src.get(start..line_end(self.src, start))?;
1132 let unquoted = text.find(['\'', '"']).unwrap_or(text.len());
1133 let k = text.get(..unquoted)?.find('\u{feff}')?;
1134 about_syntax.then_some(start.saturating_add(k))
1135 });
1136 // The mark that made this construct fail is reported
1137 // too, with its own span, as a zero-width event right after the
1138 // error (the first error event is the first diagnostic by
1139 // position, the one a strict parse fails with).
1140 //
1141 // A mark is never before `start` (= `self.pos`), so the error always
1142 // comes first. The fallback above searches from `start`;
1143 // `mark_then_fail` callers pass a mark found from the scan position on
1144 // (`directive`) or in a block scalar header that the scan position has
1145 // not passed (`block_scalar`); `block_scalar_error` places `pos` at the
1146 // offending line's start, or past its leading spaces, which hold no
1147 // mark; and a node head's mark (`head`, the mark after its content) is
1148 // used only by a failure at the head's start, or cleared (`commit_key`,
1149 // `emit_node`, `inline_colon`, a head that is not a key) before the
1150 // scan position moves past it. Never a panic: should that ever break,
1151 // an internal error in release, never a misordered report.
1152 let mark_diag =
1153 |m: usize| misplaced_bom_diagnostic(m, m.saturating_add('\u{feff}'.len_utf8()));
1154 if mark.is_some_and(|m| m < start) {
1155 debug_assert!(false, "a byte order mark before the error's start");
1156 self.internal_error("a byte order mark before the error's start");
1157 }
1158 // At a document marker the error is zero-width, so resynchronisation
1159 // resumes at that marker (R4).
1160 let at_marker = start == self.line_start
1161 && (scan::any_marker_at(self.src, start) || self.prefix_mark_at(start));
1162 let end = if at_marker {
1163 start
1164 } else {
1165 line_end(self.src, start)
1166 };
1167 // The report points at the first character that cannot belong, when
1168 // the producer named one; the event keeps covering the line.
1169 let blame = blame.or_else(|| scalar::quoted_fault(self.src, start, code));
1170 let at = blame.filter(|b| *b >= start).unwrap_or(start);
1171 // A byte order mark that made the construct fail is reported right
1172 // after it, so the error never starts after that mark.
1173 let at = mark.filter(|m| *m >= start).map_or(at, |m| at.min(m));
1174 let mut diag = Diagnostic::error(code, message, Label::new(Span::new(at, end.max(at)), ""));
1175 if let Some(help) = self.limit_help(code) {
1176 diag = diag.with_help(help);
1177 } else if let Some(help) = default_help(code) {
1178 diag = diag.with_help(help);
1179 }
1180 let diag = refine(diag);
1181 self.push(EventKind::Error(Box::new(diag)), start, end);
1182 self.pos = end;
1183 if let Some(m) = mark.filter(|m| *m >= start) {
1184 self.push(EventKind::Error(mark_diag(m)), end, end);
1185 }
1186 let mut repairs = core::mem::take(&mut self.flow_repair);
1187 for owed in repairs.drain(..) {
1188 match owed {
1189 Owed::Kind(kind) => self.zero(kind),
1190 Owed::Empty(props) => self.zero_node(EventKind::plain(), props),
1191 }
1192 }
1193 self.flow_repair = repairs;
1194 self.repair(slot_open);
1195 self.materialize();
1196 self.state = State::Skip;
1197 }
1198
1199 /// [`Parser::fail`], also reporting the byte order mark at `mark`, if
1200 /// any, that made the construct fail (after the error
1201 /// when the mark lies inside the offending line).
1202 pub(super) fn mark_then_fail(
1203 &mut self,
1204 mark: Option<usize>,
1205 code: Code,
1206 message: &str,
1207 slot_open: bool,
1208 ) {
1209 self.mark_before_error = mark;
1210 self.fail(code, message, slot_open);
1211 }
1212
1213 /// [`Parser::fail`] for a broken internal invariant: a `Y9000` error
1214 /// with the usual repair and resynchronisation. Callers
1215 /// `debug_assert!` first, so this runs only in release builds.
1216 fn fail_internal(&mut self, what: &str, slot_open: bool) {
1217 let message = format!("internal error: {what}");
1218 self.fail_with(INTERNAL_ERROR, &message, slot_open, |d| {
1219 d.with_help("this is a bug in oneyaml; please report it with this input")
1220 });
1221 }
1222
1223 /// Reports a broken internal invariant where no repair is possible: a
1224 /// zero-width `Y9000` error at `emit_pos` after any owed events, then
1225 /// the pending bytes as skipped trivia, so the stream still tiles (C1).
1226 /// Callers `debug_assert!` first.
1227 fn internal_error(&mut self, what: &str) {
1228 self.materialize();
1229 let at = self.emit_pos;
1230 self.push(EventKind::Error(internal(what, Span::empty(at))), at, at);
1231 if at < self.pos {
1232 let end = self.pos;
1233 self.push(EventKind::Trivia(TriviaKind::Skipped), at, end);
1234 }
1235 }
1236
1237 /// A zero-width error at `emit_pos` that needs no repair (the parse
1238 /// continues as if the problem were absent).
1239 fn soft_error(&mut self, code: Code, message: &str) {
1240 let at = self.emit_pos;
1241 self.soft_error_at(code, message, at);
1242 }
1243
1244 /// [`Parser::soft_error`] whose diagnostic points at `at`, an earlier
1245 /// position than the event (which stays zero-width at the scan end).
1246 fn soft_error_at(&mut self, code: Code, message: &str, at: usize) {
1247 self.zero(EventKind::Error(diagnostic(code, message, Span::empty(at))));
1248 }
1249
1250 /// Completes every open block collection with zero-width events after an
1251 /// error (C6 R2). The document stays open until resynchronisation.
1252 fn repair(&mut self, slot_open: bool) {
1253 // A failed directive block declares nothing (§6.8): its `%TAG`
1254 // handles and `%YAML` must not reach a later document.
1255 self.directives_pending = false;
1256 self.pending_tags.clear();
1257 self.yaml_seen = false;
1258 self.prefix_without_end = None;
1259 if self.doc_start_deferred {
1260 // Nothing of this document was emitted: drop it entirely.
1261 self.doc_start_deferred = false;
1262 self.deferred.clear();
1263 self.props = Properties::none();
1264 self.stack.clear();
1265 return;
1266 }
1267 if slot_open || !self.props.is_empty() {
1268 self.empty();
1269 self.child_done();
1270 }
1271 loop {
1272 match self.stack.last().copied() {
1273 Some(Frame::Map { phase, .. }) if phase != MapPhase::Key => {
1274 self.empty();
1275 self.child_done();
1276 }
1277 Some(Frame::Map { .. } | Frame::Seq { .. }) => self.close_block(false),
1278 Some(Frame::Document { root_done }) => {
1279 if !root_done {
1280 self.empty();
1281 self.child_done();
1282 }
1283 break;
1284 }
1285 None => break,
1286 }
1287 }
1288 }
1289
1290 /// Skips one line (or line break) after an error, until a document
1291 /// marker at the start of a line or the end of input (C6 R4). One line
1292 /// per step, so skipping never buffers.
1293 fn skip(&mut self) {
1294 let i = self.pos;
1295 let blen = break_len(self.src, i);
1296 if blen > 0 {
1297 let end = i.saturating_add(blen);
1298 self.push(EventKind::Trivia(TriviaKind::LineBreak), i, end);
1299 self.pos = end;
1300 self.line_start = end;
1301 self.line_indent = 0;
1302 return;
1303 }
1304 // A mark at the start of a line may start a document
1305 // prefix (production 211), so recovery resumes there too.
1306 let resync = i >= self.src.len()
1307 || !self.halted
1308 && i == self.line_start
1309 && (scan::any_marker_at(self.src, i) || scan::mark_at(self.src, i));
1310 if !resync {
1311 let end = line_end(self.src, i);
1312 self.push(EventKind::Trivia(TriviaKind::Skipped), i, end);
1313 self.pos = end;
1314 return;
1315 }
1316 self.fresh = true;
1317 self.line_indent = 0;
1318 self.tab_seen = false;
1319 self.state = if self.stack.is_empty() {
1320 State::Stream
1321 } else {
1322 State::Next
1323 };
1324 }
1325}
1326
1327#[cfg(test)]
1328mod equivalence_tests;
1329#[cfg(test)]
1330mod tests;