Crate lang_tester[][src]

Expand description

This crate provides a simple language testing framework designed to help when you are testing things like compilers and virtual machines. It allows users to express simple tests for process success/failure and for stderr/stdout, including embedding those tests directlly in the source file. It is loosely based on the compiletest_rs crate, but is much simpler (and hence sometimes less powerful), and designed to be used for testing non-Rust languages too.

For example, a Rust language tester, loosely in the spirit of compiletest_rs, looks as follows:

use std::{fs::read_to_string, path::PathBuf, process::Command};

use lang_tester::LangTester;
use tempfile::TempDir;

static COMMENT_PREFIX: &str = "//";

fn main() {
    // We use rustc to compile files into a binary: we store those binary files into `tempdir`.
    // This may not be necessary for other languages.
    let tempdir = TempDir::new().unwrap();
        // Only use files named `*.rs` as test files.
        .test_file_filter(|p| p.extension().unwrap().to_str().unwrap() == "rs")
        // Extract the first sequence of commented line(s) as the tests.
        .test_extract(|p| {
                // Skip non-commented lines at the start of the file.
                .skip_while(|l| !l.starts_with(COMMENT_PREFIX))
                // Extract consecutive commented lines.
                .take_while(|l| l.starts_with(COMMENT_PREFIX))
                // Strip the initial "//" from commented lines.
                .map(|l| &l[COMMENT_PREFIX.len()..])
        // We have two test commands:
        //   * `Compiler`: runs rustc.
        //   * `Run-time`: if rustc does not error, and the `Compiler` tests succeed, then the
        //     output binary is run.
        .test_cmds(move |p| {
            // Test command 1: Compile `` into `tempdir/x`.
            let mut exe = PathBuf::new();
            let mut compiler = Command::new("rustc");
            compiler.args(&["-o", exe.to_str().unwrap(), p.to_str().unwrap()]);
            // Test command 2: run `tempdir/x`.
            let runtime = Command::new(exe);
            vec![("Compiler", compiler), ("Run-time", runtime)]

This defines a lang tester that uses all *.rs files in a given directory as test files, running two test commands against them: Compiler (i.e. rustc); and Run-time (the compiled binary).

Users can then write test files such as the following:

// Compiler:
//   status: success
//   stderr:
//     warning: unused variable: `x`
//       ...
// Run-time:
//   status: success
//   stdout: Hello world
fn main() {
    let x = 0;
    println!("Hello world");

lang_tester is entirely ignorant of the language being tested, leaving it entirely to the user to determine what the test data in/for a file is. In this case, since we are embedding the test data as a Rust comment at the start of the file, the test_extract function we specified returns the following string:

    warning: unused variable: `x`

  stdout: Hello world

Test data is specified with a two-level indentation syntax: the outer most level of indentation defines a test command (multiple command names can be specified, as in the above); the inner most level of indentation defines alterations to the general command or sub-tests. Multi-line values are stripped of their common indentation, such that:


defines a test command x with a value a\n b\nc. Trailing whitespace is preserved.

String matching is performed by the fm crate, which provides support for ... operators and so on. Unless lang_tester is explicitly instructed otherwise, it uses fm’s defaults. In particular, even though lang_tester preserves (some) leading and (all) trailing whitespace, fm ignores leading and trailing whitespace by default (though this can be changed).

Each test command must define at least one sub-test:

  • status: <success|error|signal|<int>>, where success and error map to platform specific notions of a command completing successfully or unsuccessfully respectively. signal checks for termination due to a signal on Unix platforms; on non-Unix platforms, the test will be ignored. <int> is a signed integer checking for a specific exit code on platforms that support it. If not specified, defaults to success.
  • stderr: [<string>], stdout: [<string>] match <string> against a command’s stderr or stdout. The special string ... can be used as a simple wildcard: if a line consists solely of ..., it means “match zero or more lines”; if a line begins with ..., it means “match the remainder of the line only”; if a line ends with ..., it means “match the start of the line only”. A line may start and end with .... Note that stderr/stdout matches ignore leading/trailing whitespace and newlines, but are case sensitive. If not specified, defaults to ... (i.e. match anything). Note that the empty string matches only the empty string so e.g. stderr: on its own means that a command’s stderr muct not contain any output.

Test commands can alter the general command by specifying zero or more of the following:

  • env-var: <key>=<string> will set (or override if it is already present) the environment variable <key> to the value <string>. env-var can be specified multiple times, each setting an additional (or overriding an existing) environment variable.
  • exec-arg: <string> specifies a string which will be passed as an additional command-line argument to the command (in addition to those specified by the test_cmds function). exec-arg can be specified multiple times, each adding an additional command-line argument.
  • stdin: <string>, text to be passed to the command’s stdin. If the command exits without having consumed all of <string>, an error will be raised. Note, though, that operating system file buffers can mean that the command appears to have consumed all of <string> without it actually having done so.

The above file thus contains 4 meaningful tests, two specified by the user and two implied by defaults: the Compiler should succeed (e.g. return a 0 exit code when run on Unix), and its stderr output should warn about an unused variable on line 12; and the resulting binary should succeed produce Hello world on stdout.

A file’s tests can be ignored entirely if a test command ignore is defined:

  • ignore: [<string>], specifies that this file should be ignored for the reason set out in <string> (if any). Note that <string> is purely for user information and has no effect on the running of tests.

lang_tester’s output is deliberately similar to Rust’s normal testing output. Running the example rust_lang_tester in this crate produces the following output:

$ cargo run --example=rust_lang_tester
   Compiling lang_tester v0.1.0 (/home/ltratt/scratch/softdev/lang_tester)
    Finished dev [unoptimized + debuginfo] target(s) in 3.49s
     Running `target/debug/examples/rust_lang_tester`

running 4 tests
test lang_tests::no_main ... ok
test lang_tests::unknown_var ... ok
test lang_tests::unused_var ... ok
test lang_tests::exit_code ... ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

If you want to run a subset of tests, you can specify simple filters which use substring match to run a subset of tests:

$ cargo run --example=rust_lang_tester var
   Compiling lang_tester v0.1.0 (/home/ltratt/scratch/softdev/lang_tester)
    Finished dev [unoptimized + debuginfo] target(s) in 3.37s
     Running `target/debug/examples/rust_lang_tester var`

running 2 tests
test lang_tests::unknown_var ... ok
test lang_tests::unused_var ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out

Users will often want to integrate such tests into their test suite. An easy way of doing this is to add a [[test]] entry to your Cargo.toml along the following lines:

name = "lang_tests"
path = "lang_tests/"
harness = false

Running cargo test will now also run your lang tests.