# Architecture
## Contents
- [Markdown stream processor](#markdown-stream-processor)
- [Footnote conversion](#footnote-conversion)
- [HTML table support](#html-table-support-in-mdtablefix)
- [Module relationships](#module-relationships)
- [Concurrency with `rayon`](#concurrency-with-rayon)
- [Unicode width handling](#unicode-width-handling)
## Markdown stream processor
`process_stream_inner` orchestrates line-by-line rewriting. The full
implementation lives in [src/process.rs](../src/process.rs). Its signature is:
```rust
pub fn process_stream_inner(lines: &[String], opts: Options) -> Vec<String>
```
The function combines several helpers documented in `docs/`:
- `frontmatter::split_leading_yaml_frontmatter` detects and splits a leading
YAML frontmatter block from the document body. A valid frontmatter block
starts with `---` on the first line and ends with `---` or `...` before any
body content. The prefix is preserved verbatim while only the body is
processed. This shielding also applies to CLI-only transforms such as
`renumber_lists` and `format_breaks`.
- `fences::compress_fences` and `attach_orphan_specifiers` normalize code block
delimiters. The latter keeps indentation from the language line when the
fence lacks it. Language specifiers explicitly set to `null`
(case-insensitive) or consisting solely of whitespace are treated as absent.
`compress_fences` also tolerates spaces within comma-separated specifiers,
e.g. `TOML, Ini` becomes `toml,ini`.
- `html::convert_html_tables` transforms basic HTML tables into Markdown so \
they can be reflowed like regular tables. See \
[HTML table support](#html-table-support-in-mdtablefix).
- `wrap::wrap_text` applies optional line wrapping. It relies on the
`unicode-width` crate for accurate character widths.
- `wrap::tokenize_markdown` emits `Token` values for custom processing.
- `headings::convert_setext_headings` rewrites Setext headings with underline
markers into ATX headings when the CLI `--headings` flag is provided. The
underline must contain at least three identical `=` or `-` characters, so the
converter can distinguish headings from thematic breaks or list markers. The
helper only rewrites lines whose shared prefix is whitespace or `>` so nested
lists continue to behave normally.
Heading conversion runs after fence/table processing and before wrapping, so
the wrapping stage observes ATX headings and leaves them untouched.
The function maintains a small state machine that tracks whether it is inside a
Markdown table, an HTML table, or a fenced code block. The state determines how
incoming lines are buffered or emitted. Once the end of a table or fence is
reached, buffered lines are flushed and possibly reformatted. The simplified
behaviour is illustrated below.
````mermaid
stateDiagram-v2
[*] --> Streaming: Start
Streaming: Default state—processing lines individually
InMarkdownTable: Buffering lines of a Markdown table
InHtmlTable: Buffering lines of an HTML table
InCodeFence: Passing through lines within a fenced code block
Streaming --> InMarkdownTable: Line starts with "|"
Streaming --> InHtmlTable: Line contains table HTML tag
Streaming --> InCodeFence: Line is a fence delimiter ("```" or "~~~")
InMarkdownTable --> Streaming: Flush buffer and reflow table on non-table line (e.g., blank, heading)
InMarkdownTable --> InMarkdownTable: Line contains "|" or separator pattern
InHtmlTable --> Streaming: Flush buffer and convert table on final table HTML closing tag
InHtmlTable --> InHtmlTable: Line inside table tag
InCodeFence --> Streaming: Line is a fence delimiter
````
Before:
```markdown
|1|22|
<table><tr><td>3</td><td>4</td></tr></table>
```
After:
```markdown
| 1 | 22 |
| 3 | 4 |
```
Code fences are passed through verbatim:
```rust
After scanning all lines, the processor performs optional post-processing steps
such as ellipsis replacement and footnote conversion. See \
[footnote conversion](#footnote-conversion) for details. The function then
returns the updated stream for writing to disk or further manipulation.
## Footnote Conversion
`mdtablefix` can optionally convert bare numeric references into
GitHub-flavoured Markdown footnotes. A bare numeric reference is a number that
appears after punctuation or before a colon with no footnote formatting, for
example:
```markdown
An example of a bare numeric reference.1
The official docs page showcases several types 7:
```
`convert_footnotes` performs this operation and is exposed via the higher-level
`process_stream_opts` helper. Set
`Options { footnotes: true, ..Default::default() }` when calling
`process_stream_opts` to enable the conversion logic. The parameter defaults to
`false`.
Inline references that appear after punctuation or before a colon are rewritten
as footnote links.
Before:
```markdown
A useful tip.1
Core types 7:
```
After:
```markdown
A useful tip.[^1]
Core types```
Numbers inside inline code or parentheses are ignored.
ATX heading lines (including those nested in blockquotes and list items) are
not processed for footnote conversion, so identifiers like "A.2" remain
verbatim. Setext-style headings are not detected unless the `--headings`
conversion already rewrote them into ATX headings earlier in the pipeline.
Before:
```markdown
Look at `code 1` for details.
Refer to equation (1) for context.
```
After:
```markdown
Look at `code 1` for details.
Refer to equation (1) for context.
```
When the final lines of a document form a numbered list that is preceded by an
H2 heading (the heading text is not inspected), and the document contains no
existing footnote definitions outside fenced code blocks, they are replaced
with footnote definitions. Blank lines and indentation within the list are
tolerated; blank-only trailing blocks are ignored. Lines beginning with inline
footnote references at the start of a line do not count as existing
definitions, allowing references before the final footnote definition block.
Definitions prefixed by blockquote markers (`>`) still count as existing
blocks, but those inside fenced code blocks are ignored.
Once inline references and trailing lists are normalised, `renumber_footnotes`
walks the document in the order readers encounter references. It assigns
sequential identifiers starting from one, rewrites every reference to use its
new identifier, and updates footnote definitions to match. Trailing numeric
lists are converted into definitions when the document already contains at
least one footnote reference or definition, ensuring unrelated lists are left
untouched. The rewritten definitions are then sorted numerically so the
rendered footnote block mirrors the logical ordering of references in the text.
Before:
```markdown
Text.
## Footnotes
1. First note
2. Second note
10. Final note
```
After:
```markdown
Text.
## Footnotes
[^1]: First note
[^2]: Second note
[^10]: Final note
```
`convert_footnotes` only processes the final contiguous numeric list that
immediately follows an H2 heading when these conditions are met.
## HTML Table Support in `mdtablefix`
`mdtablefix` can format simple HTML `<table>` elements embedded in Markdown.
These HTML tables are transformed into Markdown before the main table reflow
logic runs. That preprocessing is handled by the `convert_html_tables` function.
Only straightforward tables with `<tr>`, `<th>` and `<td>` tags are detected.
Attributes and tag casing are ignored, and complex nested or styled tables are
not supported. After conversion, each HTML table is represented as a Markdown
table, so the usual reflow algorithm can align its columns consistently with
the rest of the document.
```html
<table>
<tr><th>A</th><th>B</th></tr>
<tr><td>1</td><td>2</td></tr>
</table>
```
The converter checks the first table row for `<th>` cells or for `<strong>` or
`<b>` tags inside `<td>` elements to decide whether it is a header. If no such
markers exist and the table contains multiple rows, the first row is still
treated as the header, so the Markdown output includes a separator line. This
last-resort behaviour keeps simple tables readable after conversion.
## Module Relationships
This diagram illustrates the connections between the crate's modules.
```mermaid
classDiagram
class lib {
<<module>>
}
class html {
<<module>>
+convert_html_tables()
+html_table_to_markdown() %% deprecated
}
class table {
<<module>>
+reflow_table()
+split_cells()
+SEP_RE
}
class wrap {
<<module>>
+wrap_text()
+is_fence()
+Token
+tokenize_markdown()
}
class lists {
<<module>>
+renumber_lists()
}
class breaks {
<<module>>
+format_breaks()
+THEMATIC_BREAK_LEN
}
class ellipsis {
<<module>>
+replace_ellipsis()
}
class fences {
<<module>>
+compress_fences()
+attach_orphan_specifiers()
}
class footnotes {
<<module>>
+convert_footnotes()
}
class textproc {
<<module>>
+process_tokens()
}
class process {
<<module>>
+process_stream()
+process_stream_no_wrap()
}
class io {
<<module>>
+rewrite()
+rewrite_no_wrap()
}
lib --> html
lib --> table
lib --> wrap
lib --> lists
lib --> breaks
lib --> ellipsis
lib --> fences
lib --> process
lib --> io
html ..> wrap : uses is_fence
table ..> reflow : uses parse_rows, etc.
lists ..> wrap : uses is_fence
breaks ..> wrap : uses is_fence
ellipsis ..> textproc : uses process_tokens
process ..> html : uses convert_html_tables
process ..> table : uses reflow_table
process ..> wrap : uses wrap_text, is_fence
process ..> fences : uses compress_fences, attach_orphan_specifiers
process ..> ellipsis : uses replace_ellipsis
process ..> footnotes : uses convert_footnotes
footnotes ..> wrap : uses tokenize_markdown
footnotes ..> textproc : uses push_original_token
io ..> process : uses process_stream, process_stream_no_wrap
```
The `lib` module is re-exported as the public API from the other modules. The
`wrap` module exposes the `Token` enum and `tokenize_markdown` function for
custom processing. The `ellipsis` module performs text normalization, while
`footnotes` converts bare references. The `textproc` module contains shared
token-processing helpers used by both the `ellipsis` and `footnotes` modules.
Tokenization is handled by `wrap::tokenize_markdown`, replacing the small state
machine that previously resided in `process_tokens`. The `process` module
provides streaming helpers that combine the lower-level functions. The `io`
module handles filesystem operations, delegating the text processing to
`process`.
### Tokenizer flow
The inline tokenizer iterates over the source string lazily, so no duplicate
`Vec<char>` representation is required. The following diagram summarizes the
control flow, highlighting the helpers touched during whitespace, code span,
and link handling.
```mermaid
flowchart TD
A["Input text (&str)"] --> B["Initialize tokens Vec"]
B --> C["Iterate over text by byte index"]
C --> D{"Current char is whitespace?"}
D -- Yes --> E["scan_while for whitespace"]
E --> F["collect_range and push token"]
D -- No --> G{"Current char is '`'?"}
G -- Yes --> H["Check backslash escape (has_odd_backslash_escape_bytes)"]
H -- Escaped --> I["Push '`' as token"]
H -- Not escaped --> J["scan_while for code fence"]
J --> K["Find closing fence, collect_range and push token"]
G -- No --> L{"Current char is '[' or '!['?"}
L -- Yes --> M["parse_link_or_image"]
M --> N["Push link/image token"]
N --> O["scan_while for trailing punctuation"]
O --> P["collect_range and push punctuation token"]
L -- No --> Q["scan_while for non-whitespace/non-` chars"]
Q --> R["collect_range and push token"]
F & I & K & P & R --> S["Continue iteration"]
S --> C
C -->|End| T["Return tokens Vec"]
```
The helper `html_table_to_markdown` is retained for backward compatibility but
is deprecated. New code should call `convert_html_tables` instead.
## Concurrency with `rayon`
`mdtablefix` uses the `rayon` crate to process multiple files concurrently.
`rayon` provides a work-stealing thread pool and simple parallel iterators. The
tool relies on Rayon's global thread pool so that no manual setup is required.
The dependency is specified as `1.0` in `Cargo.toml` to track stable API
changes within the same major release.
Parallelism is enabled automatically whenever more than one file path is
provided on the command line. Each worker gathers its output before printing,
so results appear in the original order. This buffering increases memory usage
and may reduce performance if many tiny files are processed.
```mermaid
sequenceDiagram
participant User as actor User
participant CLI as CLI Main
participant FileHandler as handle_file
participant Stdout as Stdout
participant Stderr as Stderr
User->>CLI: Run CLI with multiple files (not in-place)
CLI->>FileHandler: handle_file(file1)
CLI->>FileHandler: handle_file(file2)
CLI->>FileHandler: handle_file(file3)
Note over CLI,FileHandler: Files processed in parallel
FileHandler-->>CLI: Result (Ok(Some(output)) or Err(error))
loop For each file in input order
CLI->>Stdout: Print output (if Ok)
CLI->>Stderr: Print error (if Err)
end
CLI-->>User: Exit (with error if any file errored)
```
## Unicode Width Handling
`mdtablefix` wraps paragraphs and list items while respecting the display width
of Unicode characters. The `unicode-width` crate is used to compute the width
of strings when deciding where to break lines. This prevents emojis or other
multibyte characters from causing unexpected wraps or truncation.
Whenever wrapping logic examines the length of a token, it relies on
`UnicodeWidthStr::width` to measure visible columns rather than byte length.
## Link punctuation handling
Trailing punctuation immediately following a Markdown link or image is
tokenized separately and grouped with the link when wrapping. This keeps
sentences like:
```markdown
[link](path).
```
on a single line, rather than splitting the punctuation onto the next line when
wrapping occurs.
## Inline code punctuation handling
Trailing punctuation that follows an inline code span is grouped with the code
when wrapping. This prevents sentences such as:
```markdown
`useState`.
```
from splitting the full stop onto a new line, preserving the code span's
readability.