A test runner that builds tests with rustc or cargo (or any other compiler
with some configuration effort) and compares the output of the compiler with
a file that you check into git. This allows you to test how your libraries
show up to your users when the library is used wrongly and emits errors.
## Usage
See [examples directory](examples) for how to use this in your own crate.
To be able to use it with `cargo test`, you need to put
```toml
[[test]]
name = "your_test_file"
harness = false
```
into your `Cargo.toml`, otherwise `cargo test` will only look for `#[test]`s and
not run your `fn main()` that actually executes `ui_test`
## Implicit (and possibly surprising) behavior
* Tests are run in order of their filenames (files first, then recursing into folders).
So if you have any slow tests, prepend them with a small integral number to make them get run first, taking advantage of parallelism as much as possible (instead of waiting for the slow tests at the end).
* `cargo test --test your_test_name -- --help` lists the commands you can specify for filtering, blessing and making your tests less verbose.
* Since `cargo test` on its own runs all tests, using `cargo test -- --check` will not work on its own, but `cargo test -- --quiet` and `cargo test -- some_test_name` will work just fine, as the CLI matches.
* if there is a `.stdin` file with the same filename as your test, it will be piped as standard input to your program.
## Supported comment annotations
If your test tests for failure, you need to add a `//~` annotation where the error is happening
to ensure that the test will always keep failing at the annotated line. This will match any substring
of the error. These comments can take two forms:
* `//~ LEVEL: XXX` matches by error level and message text
* `LEVEL` can be one of the following (descending order): `ERROR`, `HELP`, `WARN`, `NOTE` or `ICE`
* If a level is specified explicitly, *all* diagnostics of that level or higher need an annotation. To avoid this see `//@require-annotations-for-level`
* This checks the output *before* normalization, so you can check things that get normalized away, but need to
be careful not to accidentally have a pattern that differs between platforms.
* if `XXX` is of the form `/XXX/` it is treated as a regex instead of a substring and will succeed if the regex matches.
* `//~ CODE` matches by diagnostic code.
* `CODE` can take multiple forms such as: `E####`, `lint_name`, `tool::lint_name`.
* This will only match a diagnostic at the `ERROR` level.
The annotation can be put on a different line from the error location using `v`/`^`:
```rs
causes_some_error()
//~^ some_error
//~v some_error
causes_some_error()
```
By repeating those symbols `n` times, one can annotate an error located `n` lines away:
```rs
causes_some_error()
//~^^ some_error
```
If there are multiple errors on the same line, annotations can be put on consecutive lines, and "chained" with `|`:
```rs
causes_some_error_and_other_error()
//~^ some_error
## Significant differences to compiletest-rs
* target selection is done via `ignore-target: xxx` and `only-target: xxx` instead of compiletest's
`ignore-xxx`/`only-xxx`. The `xxx` must also be a substring of the target triple; special
collections such as `macos`/`unix` in compiletest is not supported.
* only supports `ui` tests
* tests are run in named order, so you can prefix slow tests with `0` in order to make them get run first
* `aux-build`s require specifying nested aux builds explicitly and will not allow you to reference sibling `aux-build`s' artifacts.