# lang_tester
This crate provides a simple language testing framework designed to help when
you are testing things like compilers and virtual machines. It allows users to
express simple tests for process success/failure and for stderr/stdout, including
embedding those tests directlly in the source file. It is loosely based on the
[`compiletest_rs`](https://crates.io/crates/compiletest_rs) crate, but is much
simpler (and hence sometimes less powerful), and designed to be used for
testing non-Rust languages too.
For example, a Rust language tester, loosely in the spirit of
[`compiletest_rs`](https://crates.io/crates/compiletest_rs), looks as follows:
```rust
use std::{fs::read_to_string, path::PathBuf, process::Command};
use lang_tester::LangTester;
use tempfile::TempDir;
static COMMENT_PREFIX: &str = "//";
fn main() {
// We use rustc to compile files into a binary: we store those binary files
// into `tempdir`. This may not be necessary for other languages.
let tempdir = TempDir::new().unwrap();
LangTester::new()
.test_dir("examples/rust_lang_tester/lang_tests")
// Only use files named `*.rs` as test files.
.test_file_filter(|p| p.extension().unwrap().to_str().unwrap() == "rs")
// Extract the first sequence of commented line(s) as the tests.
.test_extract(|p| {
read_to_string(p)
.unwrap()
.lines()
// Skip non-commented lines at the start of the file.
.skip_while(|l| !l.starts_with(COMMENT_PREFIX))
// Extract consecutive commented lines.
.take_while(|l| l.starts_with(COMMENT_PREFIX))
.map(|l| &l[COMMENT_PREFIX.len()..])
.collect::<Vec<_>>()
.join("\n")
})
// We have two test commands:
// * `Compiler`: runs rustc.
// * `Run-time`: if rustc does not error, and the `Compiler` tests
// succeed, then the output binary is run.
.test_cmds(move |p| {
// Test command 1: Compile `x.rs` into `tempdir/x`.
let mut exe = PathBuf::new();
exe.push(&tempdir);
exe.push(p.file_stem().unwrap());
let mut compiler = Command::new("rustc");
compiler.args(&["-o", exe.to_str().unwrap(), p.to_str().unwrap()]);
// Test command 2: run `tempdir/x`.
let runtime = Command::new(exe);
vec![("Compiler", compiler), ("Run-time", runtime)]
})
.run();
}
```
This defines a lang tester that uses all `*.rs` files in a given directory as
test files, running two test commands against them: `Compiler` (i.e. `rustc`);
and `Run-time` (the compiled binary).
Users can then write test files such as the following:
```rust
// Compiler:
// stderr:
// warning: unused variable: `x`
// ...unused_var.rs:12:9
// ...
//
// Run-time:
// stdout: Hello world
fn main() {
let x = 0;
println!("Hello world");
}
```
`lang_tester` is entirely ignorant of the language being tested, leaving it
entirely to the user to determine what the test data in/for a file is. In this
case, since we are embedding the test data as a Rust comment at the start of
the file, the `test_extract` function we specified returns the following
string:
```
Compiler:
stderr:
warning: unused variable: `x`
...unused_var.rs:12:9
...
Run-time:
stdout: Hello world
```
Test data is specified with a two-level indentation syntax: the outer most
level of indentation defines a test command (multiple command names can be
specified, as in the above); the inner most level of indentation defines
alterations to the general command or sub-tests. Multi-line values are stripped
of their common indentation, such that:
```text
x:
a
b
c
```
defines a test command `x` with a value `a\n b\nc`. Trailing whitespace
is preserved.
String matching is performed by the [fm crate](https://crates.io/crates/fm),
which provides support for `...` operators and so on. Unless `lang_tester` is
explicitly instructed otherwise, it uses `fm`'s defaults. In particular, even
though `lang_tester` preserves (some) leading and (all) trailing whitespace,
`fm` ignores leading and trailing whitespace by default (though this can be
changed).
Each test command must define at least one sub-test:
* `status: <success|error|signal|<int>>`, where `success` and `error` map
to platform specific notions of a command completing successfully or
unsuccessfully respectively. `signal` checks for termination due to a signal
on Unix platforms; on non-Unix platforms, the test will be ignored. `<int>`
is a signed integer checking for a specific exit code on platforms that
support it. If not specified, defaults to `success`.
* `stderr: [<string>]`, `stdout: [<string>]` match `<string>` against a
command's `stderr` or `stdout`. The special string `...` can be used as a
simple wildcard: if a line consists solely of `...`, it means "match zero
or more lines"; if a line begins with `...`, it means "match the remainder
of the line only"; if a line ends with `...`, it means "match the start of
the line only". A line may start and end with `...`. Note that
`stderr`/`stdout` matches ignore leading/trailing whitespace and newlines,
but are case sensitive. If not specified, defaults to `...` (i.e. match
anything). Note that the empty string matches only the empty string so
e.g. `stderr:` on its own means that a command's `stderr` muct not contain
any output.
Test commands can alter the general command by specifying zero or more of the
following:
* `env-var: <key>=<string>` will set (or override if it is already present)
the environment variable `<key>` to the value `<string>`. `env-var` can be
specified multiple times, each setting an additional (or overriding an
existing) environment variable.
* `exec-arg: <string>` specifies a string which will be passed as an
additional command-line argument to the command (in addition to those
specified by the `test_cmds` function). Multiple `exec-arg`s can be
specified, each adding an additional command-line argument.
* `stdin: <string>`, text to be passed to the command's `stdin`. If the
command exits without having consumed all of `<string>`, an error will be
raised. Note, though, that operating system file buffers can mean that the
command *appears* to have consumed all of `<string>` without it actually
having done so.
The above file thus contains 4 meaningful tests, two specified by the user and
two implied by defaults: the `Compiler` should succeed (e.g. return a `0` exit
code when run on Unix), and its `stderr` output should warn about an unused
variable on line 12; and the resulting binary should succeed produce `Hello
world` on `stdout`.
A file's tests can be ignored entirely if a test command `ignore` is defined:
* `ignore: [<string>]`, specifies that this file should be ignored for the
reason set out in `<string>` (if any). Note that `<string>` is purely for
user information and has no effect on the running of tests.
`lang_tester`'s output is deliberately similar to Rust's normal testing output.
Running the example `rust_lang_tester` in this crate produces the following
output:
```text
$ cargo run --example=rust_lang_tester
Compiling lang_tester v0.1.0 (/home/ltratt/scratch/softdev/lang_tester)
Finished dev [unoptimized + debuginfo] target(s) in 3.49s
Running `target/debug/examples/rust_lang_tester`
running 4 tests
test lang_tests::no_main ... ok
test lang_tests::unknown_var ... ok
test lang_tests::unused_var ... ok
test lang_tests::exit_code ... ok
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
```
If you want to run a subset of tests, you can specify simple filters which use
substring match to run a subset of tests:
```text
$ cargo run --example=rust_lang_tester var
Compiling lang_tester v0.1.0 (/home/ltratt/scratch/softdev/lang_tester)
Finished dev [unoptimized + debuginfo] target(s) in 3.37s
Running `target/debug/examples/rust_lang_tester var`
running 2 tests
test lang_tests::unknown_var ... ok
test lang_tests::unused_var ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out
```
## Integration with Cargo.
Tests created with lang_tester can be used as part of an existing test suite and
can be run with the `cargo test` command. For example, if the Rust source file
that runs your lang tests is `lang_tests/run.rs` then add the following to your
Cargo.toml:
```
[[test]]
name = "lang_tests"
path = "lang_tests/run.rs"
harness = false
```