Module testing 

Testing utilities for AST assertions

This module provides comprehensive testing tools and guidelines for the lex parser.
Testing the parser must follow strict rules to ensure reliability and maintainability.

Why Testing is Different

Lex is a novel format, for which there is no established body of source text nor a
reference parser to compare against. Adding insult to injury, the format is still
evolving, so specs change, and in some ways it looks like markdown just enough to
create confusion.

The corollary here being that getting correct Lex source text is not trivial, and if
you make one up, the odds of it being slightly off are high. If one tests the parser
against an illegal source string, all goes to waste: we will have a parser tuned to
the wrong thing. Worst of all, as each test might produce its slight variation, we
will have an unpredictable, complex and wrong parser. If that was not enough, come a
change in the spec, and now we must hunt down and review hundreds of ad-hoc strings
in test files.

This is why all testing must follow two strict rules:

    1. Always use verified sample files from the spec (via [Lexplore](lexplore))
    2. Always use comprehensive AST assertions (via [assert_ast](fn@assert_ast))

Rule 1: Always Use Lexplore for Test Content

Why this matters:

    lex is a novel format that's still evolving. People regularly get small details
    wrong, leading to false positives in tests. When lex changes, we need to verify
    and update all source files. If lex content is scattered across many test files,
    this becomes a maintenance nightmare.

The solution:

    Use the `Lexplore` library to access verified, curated lex sample files. This
    ensures only vetted sources are used and makes writing tests much easier.

Examples:

```rust,ignore
use crate::lex::testing::lexplore::Lexplore;
use crate::lex::parsing::parse_document;

// CORRECT: Use verified sample files
let doc = Lexplore::paragraph(1).parse().unwrap();
let paragraph = doc.root.expect_paragraph();

// OR load source and parse separately
let source = Lexplore::paragraph(1).source();
let doc = parse_document(&source).unwrap();

// OR use tokenization
let tokens = Lexplore::list(1).tokenize().unwrap();

// OR load documents (benchmark, trifecta)
let doc = Lexplore::benchmark(10).parse().unwrap();
let doc = Lexplore::trifecta(0).parse().unwrap();

// OR get the AST node directly
let paragraph = Lexplore::get_paragraph(1);
let list = Lexplore::get_list(1);
let session = Lexplore::get_session(1);

// WRONG: Don't write lex content directly in tests
let doc = parse_document("Some paragraph\n\nAnother paragraph\n\n").unwrap();
```

Available sources:

    - Elements: `Lexplore::paragraph(1)`, `Lexplore::list(1)`, etc. - Individual elements
    - Documents: `Lexplore::benchmark(0)`, `Lexplore::trifecta(0)` - Full documents
    - Direct access: `Lexplore::get_paragraph(1)` - Returns the AST node directly

The sample files are organized:

    - By elements:
        - Isolated elements (only the element itself): Individual test cases
        - In Document: mixed with other elements: Integration test cases
    - Benchmark: full documents that are used to test the parser
    - Trifecta: a mix of sessions, paragraphs and lists, the structural elements

See the [Lexplore documentation](lexplore) for complete API details.

Rule 2: Always Use assert_ast for AST Verification

Why this matters:

What we want for every document test is to ensure that the AST shape is correct per the grammar, that all attributes are correct (children, content, etc.). Asserting generalities like node counts is useless - it’s not informative. We want assurance on the AST shape and content.

This is also very hard to write, time-consuming, and when the lex spec changes, very hard to update.

The solution:

Use the assert_ast library with its fluent API. It allows testing entire hierarchies of nodes at once with 10-20x less code.

The Problem with Manual Testing

Testing a nested session traditionally looks like this:

use crate::lex::ast::ContentItem;

match &doc.content[0] {
    ContentItem::Session(s) => {
        assert_eq!(s.title, "Introduction");
        assert_eq!(s.children.len(), 2);
        match &s.content[0] {
            ContentItem::Paragraph(p) => {
                assert_eq!(p.lines.len(), 1);
                assert!(p.lines[0].starts_with("Hello"));
            }
            _ => panic!("Expected paragraph"),
        }
        // ... repeat for second child
    }
    _ => panic!("Expected session"),
}

20+ lines of boilerplate. Hard to see what’s actually being tested.

The Solution: Fluent Assertion API

With the assert_ast fluent API, the same test becomes:

use crate::lex::testing::assert_ast;

assert_ast(&doc)
    .item(0, |item| {
        item.assert_session()
            .label("Introduction")
            .child_count(2)
            .child(0, |child| {
                child.assert_paragraph()
                    .text_starts_with("Hello")
            })
    });

Concise, readable, and maintainable.

Available Node Types

The assertion API supports all AST node types:

• ParagraphAssertion - Text content nodes
• SessionAssertion - Titled container nodes
• ListAssertion / ListItemAssertion - List structures
• DefinitionAssertion - Subject-definition pairs
• AnnotationAssertion - Metadata with parameters
• VerbatimBlockkAssertion - Raw content blocks Each assertion type provides type-specific methods (e.g., label() for sessions, subject() for definitions, parameter_count() for annotations).

Extending the Assertion API

To add support for a new container node type:

1. Implement the traits in ast.rs:

   use crate::lex::ast::{Container, ContentItem};
   
   struct NewNode { content: Vec<ContentItem>, label: String }
   
   impl Container for NewNode {
       fn label(&self) -> &str { &self.label }
       fn children(&self) -> &[ContentItem] { &self.content }
       fn children_mut(&mut self) -> &mut Vec<ContentItem> { &mut self.content }
   }

2. Add to ContentItem enum and implement helper methods

3. Add assertion type in testing_assertions.rs:

   pub struct NewNodeAssertion<'a> { /* ... */ }
   
   impl NewNodeAssertion<'_> {
       pub fn custom_field(self, expected: &str) -> Self { /* ... */ }
       pub fn child_count(self, expected: usize) -> Self { /* ... */ }
   }

4. Add to ContentItemAssertion and export in testing.rs:

   pub fn assert_new_node(self) -> NewNodeAssertion<'a> { /* ... */ }

Modules

factories

lexplore

Test harness for per-element testing

text_diff

Line-based text diffing utilities for testing

Structs

AnnotationAssertion

ChildrenAssertion

ContentItemAssertion

DefinitionAssertion

DocumentAssertion

InlineAssertion

InlineExpectation

ListAssertion

ListItemAssertion

ParagraphAssertion

ReferenceExpectation

SessionAssertion

VerbatimBlockkAssertion

Enums

TextMatch

Text matching strategies for assertions

Functions

assert_ast

Create an assertion builder for a document

parse_without_annotation_attachment

Parse a Lex document without running the annotation attachment stage.

workspace_path

Get a path relative to the crate root for testing purposes.