This is a tool for processing Markdown files with refactoring commands embedded
in them. It runs any refactoring commands it encounters, and inserts a
rendered diff of the results after each refactoring block.
Basic usage:
python3 -m literate --project-dir repo/rust render input.md output.md
The tool is controlled mainly by specially-tagged code blocks in the Markdown
input.
## Refactoring commands
Most importantly, blocks with their language set to `refactor` are interpreted
as refactoring commands:
```refactor
rewrite_expr '1 + 1' '2'
```
For each such block, the block itself is included in the output (with language
changed to `sh`, since refactor commands and arguments are parsed as shell
words), followed by a diff showing the effect it had on the code being
refactored.
## Options
In each code block header, the language can be followed by space-separated
options. This tool supports the following options:
* `refactor-target`: Sets the contents of the current block as the current
crate. Future `refactor` blocks will rewrite this code instead of the
contents of the `--project-dir`.
* `hidden`: The code block is not included in the output. This allows
including a `refactor-target` or `refactor-options` block only for its side
effects, without display it to the reader.
* `revert`: After processing the block, discard the results of refactoring,
reverting the crate to its state before the block. This is only meaningful
on `refactor` blocks. This is useful for showing the effects of several
different refactoring commands on the same initial code.
* `filename`: Display the filenames of rewritten files. Default: `true`.
Disabling this is useful in combination with `refactor-target`, which causes
refactoring to run on temporary files with generated names.
* `collapse-diff`: Put the diff inside a `<details>` tag, causing browsers to
display it collapsed until the reader clicks on it. Default: `true`.
* `hide-diff`: Omit the diff entirely, while still including the code block in
the output.
* `hide-code`: Omit the code block entirely, while still including the diff in
the output.
* `diff-style = full|context|only-new': Sets the diff rendering mode. This is
only meaningful on `refactor` blocks. The default is `context`, which
renders only changed lines and nearby context. `full` renders the entire
contents of both files, which is useful for tutorials, where the files being
refactored are usually very small. `only-new` renders only the new file,
not the old one (and omits insertion/deletion markers).
Boolean options can be prefixed with `no-` to disable them. For example,
`no-hidden` cancels the effect of a previous `hidden`, causing the block to
once again appear in the output.
Since code block headers are split on whitespace, attributes with values must
not contain space around the `=`. Write `diff-style=full`, not `diff-style =
full`.
## Global options
Options can be set globally by including a block whose language is
`refactor-options`:
```refactor-options hidden
# This is a comment
diff-style = full
revert
```
Any options set this way will be applied to all code blocks in the remainder of
the file. In this example, all later refactoring blocks will produce full
diffs, and will revert changes after processing. Global options can still be
overridden on specific blocks, so a block whose header reads `refactor
diff-style=context no-revert` would work as before.
# Internals
The central data structure of `literate.py` is the `File` class, defined in
`file.py`. A `File` initially contains only basic information, like the raw
file contents, but it is incrementally updated and annotated with additional
information. Part of this process involves constructing a `Diff` from two
files, which provides information needed for additional annotation and eventual
rendering.
The other essential data structures are lists of `Span`s (`annot.py`) and lists
of `Point`s (`points.py`), which are used in various ways to annotate `File`s
and their `Line`s.
The overall process looks like this (drawn from `__init__.do_render`,
`render.prepare_files`, and `render.make_diff`):
* Input parsing, which separates code blocks from non-code text (`parse.py`)
* Execution of the refactoring commands and processing of their outputs
(`refactor.py`). This is the point where the `File` objects are created,
based on the refactoring outputs.
* Formatting of source files (`format.py`). `c2rust-refactor` makes no
attempt to produce nicely-formatted output, so we need to run `rustfmt` on
the code to produce readable diffs.
* Syntax highlighting (`highlight.py`). This mostly amounts to invoking
`pygments` and annotating the file with the results.
* Mark processing, to annotate files with the positions and information about
marked nodes (`marks.py`).
* `Diff` construction, for each matching pair of old and new files
(`diff.py`). This step is partially interleaved with mark processing, as
some parts of mark processing require access to both files, while some parts
of diff construction depend on the locations of marks. See
`render.make_diff` for specifics.
* Rendering of the final diff (`render.py`).