Crate lex_babel 

Multi-format interoperability for Lex documents

This crate provides a uniform interface for converting between Lex AST and various document
formats (Markdown, HTML, Pandoc JSON, etc.).

TLDR: For format authors:
    - Babel never parses or serializes any format, but instead relies on the format's libraries
    - The conversion should be by converting to the IR, running the common code in common if relevant (it usually is), then to the AST of the target format.
    - We should use the testing harness (see lex-core/src/lex/testing.rs) to load documents and process them into ASTs.
    - Each element should use the harness above and the available file for isolated element testing with unit tests (load with the lib, assert with AST / IR)
    - Each format should have trifecta unit tested in from and to formats to lex.
    - Each format should have a kitchensink unit tested in from and to formats to lex
    - Read the README.lex for full details

Architecture

The goal here is to, as much as possible, split what is the common logic for multiple formats
conversions into a format agnostic layer. This is done by using the IR representation (./ir/mod.rs),
and having the common code in ./common/mod.rs. This allows for the format specific code to be focused on the data format transformations, while having a strong, focused core that can be well tested in isolation.

This is a pure lib, that is, it powers the lexd CLI but is shell agnostic, that is no code
should be written that supposes a shell environment, be it to std print, env vars etc.

The file structure:
.
├── error.rs
├── format.rs               # Format trait definition
├── registry.rs             # FormatRegistry for discovery and selection
├── formats
│   └── <format>
│       ├── parser.rs       # Parser implementation
│       ├── serializer.rs   # Serializer implementation
│       └── mod.rs
├── lib.rs
├── ir                      # Intermediate Representation
│   ├── nodes.rs            # IR data model (DocNode, InlineContent, etc.)
│   ├── events.rs           # Flat event stream representation
│   ├── from_lex.rs         # Lex AST → IR conversion
│   └── to_lex.rs           # IR → Lex AST conversion
└── common                  # Common mapping code
    ├── flat_to_nested.rs   # Events → IR tree (with auto-closing)
    ├── nested_to_flat.rs   # IR tree → Events
    └── verbatim/           # Verbatim handler registry (doc.table, doc.image, etc.)

Testing tests └── ├── .rs └── fixtures ├── . ├── kitchensink.html ├── kitchensink.lex └── kitchensink.md

Note that rust does not by default discover tests in subdirectories, so we need to include these
in the mod.

Core Algorithms

The most complex part of the work is reconstructing a nested representation from a flat document, followed by the reverse operations. For this reason we have a common IR (./ir/mod.rs) that is used for all formats.
Over this representation we implement both algorithms (see ./common/flat_to_nested.rs and ./common/nested_to_flat.rs).
This means that all the heavy lifting is done by a core, well tested and maintained module,
freeing format adaptations to be focused on the simpler data format transformations.

Formats

Format specific capabilities are implemented with the Format trait. Formats should have a
parse() and serialize() method, a name and file extensions. See the trait def [./format.rs]
- Format trait: Uniform interface for all formats (parsing and/or serialization)
- FormatRegistry: Centralized discovery and selection of formats
- Format implementations: Concrete implementations for each supported format

The Lex Format

The Lex format itself is implemented as a format, see ./formats/lex/mod.rs, which allows for
a homogeneous API where all formats have identical interfaces.

Note that Lex is a more expressive format than most, which means that converting from Lex is
simple, but always lossy. In particular converting to Lex requires some consideration on how
to best represent the author's intent.

This means that full format interop round tripping is not possible.

v1 Interop Scope

The full contributor-facing version of this lives in
`comms/docs/interop-scope.lex` (browse on GitHub:
<https://github.com/lex-fmt/comms/blob/main/docs/interop-scope.lex>).
The short version:

| Tier              | Format     | Export | Import | Notes                                                |
|-------------------|------------|--------|--------|------------------------------------------------------|
| Core              | Markdown   | ✓      | ✓      | Lingua franca. Round-trip is the bar.                |
| Core              | HTML       | ✓      | —      | Publishing target; PDF and editor previews consume.  |
| Core              | PDF        | ✓      | —      | Headless Chrome over the HTML output.                |
| Core              | PNG        | ✓      | —      | Headless Chrome screenshot of the HTML output.       |
| Stretch           | HTML       | —      | ✓      | After core lands.                                    |
| Experimental      | RFC XML    | —      | ✓      | Proof-of-concept; no bespoke investment.             |
| Planned           | Pandoc     | —      | —      | Bridge to DOCX/EPUB/RST/Org/etc. Not started.        |
| Planned           | LaTeX      | ✓      | —      | Export only, via Pandoc once Pandoc lands.           |
| Category error    | PDF import | —      | —      | **Will not be implemented.** See below.              |

**PDF import is a category error, not a postponed task.** PDF is a
presentation format — paragraphs, headings, and lists are
reconstructed heuristically from layout (font size, indentation,
glyph positions), and no rule-based importer recovers that
reliably. Pandoc punts to `pdftotext` for the same reason. This
will not be implemented, ever, regardless of adoption. ML-based
extraction (Marker, Nougat, Mathpix, Grobid) is a different
product category and out of scope for lex-babel. Do not list,
advertise, or design around the possibility.

Diagnostic outputs (`tag`, `treeviz`, `linetreeviz`) sit outside
this tiering — they're AST visualizers used by `lexd inspect`,
not interop targets.

Library Choices

This, not being Lex's core, means that we will offload as much as possible to better, specialized crates
for each format. The scope here is mainly to adapt the ASTs from Lex to the format or vice
versa. For example we never write the serializer for, say, Markdown, but pass the AST to the
Markdown library. To support a format inbound, we write the format AST → Lex AST adapter.
Likewise, for outbound formats we will do the reverse, converting from the Lex AST to the
format's.

As much as possible, we will use Rust crates, and avoid shelling out and having outside dependencies.

Re-exports

pub use error::FormatError;

pub use format::Format;

pub use format::SerializedDocument;

pub use registry::FormatRegistry;

Modules

common

Contains logic for mapping between different document representations.

error

Error types for format operations

format

Format trait definition

formats

Format implementations

ir

Intermediate Representation (IR) for lex documents.

publish

Document publishing pipeline.

registry

Format registry for format discovery and selection

render_dispatch

Render-hook dispatch: walk the IR, ask registered handlers to render their labelled annotations / verbatim blocks, and surface the results to the format-specific serializer.

templates

Ready-to-insert snippets for Lex documents.

transforms

Transform integration for lex-babel formats

Functions

default_registry

Process-wide registry with the built-in lex.* schemas registered. Used by to_ir and to_lex_document for callers that don’t supply their own. Constructed lazily on first call.

from_ir

Converts an IR document back to Lex AST.

to_ir

Converts a lex document to the Intermediate Representation (IR).

to_ir_with_registry

Converts a Lex AST to its IR representation, dispatching verbatim labels through the supplied registry’s on_resolve hooks.