1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228
//! This crate provides a simple language testing framework designed to help when you are testing
//! things like compilers and virtual machines. It allows users to express simple tests for process
//! success/failure and for stderr/stdout, including embedding those tests directly in the source
//! file. It is loosely based on the [`compiletest_rs`](https://crates.io/crates/compiletest_rs)
//! crate, but is much simpler (and hence sometimes less powerful), and designed to be used for
//! testing non-Rust languages too.
//!
//! For example, a Rust language tester, loosely in the spirit of
//! [`compiletest_rs`](https://crates.io/crates/compiletest_rs), looks as follows:
//!
//! ```rust,ignore
//! use std::{env, fs::read_to_string, path::PathBuf, process::Command};
//!
//! use lang_tester::LangTester;
//! use tempfile::TempDir;
//!
//! fn main() {
//! // We use rustc to compile files into a binary: we store those binary files
//! // into `tempdir`. This may not be necessary for other languages.
//! let tempdir = TempDir::new().unwrap();
//! LangTester::new()
//! .test_dir("examples/rust_lang_tester/lang_tests")
//! // Only use files named `*.rs` as test files.
//! .test_path_filter(|p| p.extension().and_then(|x| x.to_str()) == Some("rs"))
//! // Treat lines beginning with "#" inside a test as comments.
//! .comment_prefix("#")
//! // Extract the first sequence of commented line(s) as the tests.
//! .test_extract(|p| {
//! read_to_string(p)
//! .unwrap()
//! .lines()
//! // Skip non-commented lines at the start of the file.
//! .skip_while(|l| !l.starts_with("//"))
//! // Extract consecutive commented lines.
//! .take_while(|l| l.starts_with("//"))
//! .map(|l| &l[COMMENT_PREFIX.len()..])
//! .collect::<Vec<_>>()
//! .join("\n")
//! })
//! // We have two test commands:
//! // * `Compiler`: runs rustc.
//! // * `Run-time`: if rustc does not error, and the `Compiler` tests
//! // succeed, then the output binary is run.
//! .test_cmds(move |p| {
//! // Test command 1: Compile `x.rs` into `tempdir/x`.
//! let mut exe = PathBuf::new();
//! exe.push(&tempdir);
//! exe.push(p.file_stem().unwrap());
//! let mut compiler = Command::new("rustc");
//! compiler.args(&["-o", exe.to_str().unwrap(), p.to_str().unwrap()]);
//! // Test command 2: run `tempdir/x`.
//! let runtime = Command::new(exe);
//! vec![("Compiler", compiler), ("Run-time", runtime)]
//! })
//! .run();
//! }
//! ```
//!
//! This defines a lang tester that uses all `*.rs` files in a given directory as test files,
//! running two test commands against them: `Compiler` (i.e. `rustc`); and `Run-time` (the compiled
//! binary).
//!
//! Users can then write test files such as the following:
//!
//! ```rust,ignore
//! // Compiler:
//! // stderr:
//! // warning: unused variable: `x`
//! // ...unused_var.rs:12:9
//! // ...
//! //
//! // Run-time:
//! // stdout: Hello world
//! fn main() {
//! let x = 0;
//! println!("Hello world");
//! }
//! ```
//!
//! `lang_tester` is entirely ignorant of the language being tested, leaving it entirely to the
//! user to determine what the test data in/for a file is. In this case, since we are embedding the
//! test data as a Rust comment at the start of the file, the `test_extract` function we specified
//! returns the following string:
//!
//! ```text
//! Compiler:
//! stderr:
//! warning: unused variable: `x`
//! ...unused_var.rs:12:9
//! ...
//!
//! Run-time:
//! stdout: Hello world
//! ```
//!
//! Test data is specified with a two-level indentation syntax: the outer most level of indentation
//! defines a test command (multiple command names can be specified, as in the above); the inner
//! most level of indentation defines alterations to the general command or sub-tests. Multi-line
//! values are stripped of their common indentation, such that:
//!
//! ```text
//! x:
//! a
//! b
//! c
//! ```
//!
//! defines a test command `x` with a value `a\n b\nc`. Trailing whitespace is preserved.
//!
//! String matching is performed by the [fm crate](https://crates.io/crates/fm), which provides
//! support for `...` operators and so on. Unless `lang_tester` is explicitly instructed otherwise,
//! it uses `fm`'s defaults. In particular, even though `lang_tester` preserves (some) leading and
//! (all) trailing whitespace, `fm` ignores leading and trailing whitespace by default (though this
//! can be changed).
//!
//! Each test command must define at least one sub-test:
//!
//! * `status: <success|error|signal|<int>>`, where `success` and `error` map to platform
//! specific notions of a command completing successfully or unsuccessfully respectively.
//! `signal` checks for termination due to a signal on Unix platforms; on non-Unix platforms,
//! the test will be ignored. `<int>` is a signed integer checking for a specific exit code on
//! platforms that support it. If not specified, defaults to `success`.
//! * `stderr: [<string>]`, `stdout: [<string>]` match `<string>` against a command's `stderr` or
//! `stdout`. The special string `...` can be used as a simple wildcard: if a line consists
//! solely of `...`, it means "match zero or more lines"; if a line begins with `...`, it means
//! "match the remainder of the line only"; if a line ends with `...`, it means "match the
//! start of the line only". A line may start and end with `...`. Note that `stderr`/`stdout`
//! matches ignore leading/trailing whitespace and newlines, but are case sensitive. If not
//! specified, defaults to `...` (i.e. match anything). Note that the empty string matches only
//! the empty string so e.g. `stderr:` on its own means that a command's `stderr` muct not
//! contain any output.
//!
//! Test commands can alter the general command by specifying zero or more of the following:
//!
//! * `env-var: <key>=<string>` will set (or override if it is already present) the environment
//! variable `<key>` to the value `<string>`. `env-var` can be specified multiple times, each
//! setting an additional (or overriding an existing) environment variable.
//! * `exec-arg: <string>` specifies a string which will be passed as an additional command-line
//! argument to the command (in addition to those specified by the `test_cmds` function).
//! Multiple `exec-arg`s can be specified, each adding an additional command-line argument.
//! * `stdin: <string>` specifies text to be passed to the command's `stdin`. If the command
//! exits without consuming all of `<string>`, an error will be raised. Note, though, that
//! operating system file buffers can mean that the command *appears* to have consumed all of
//! `<string>` without it actually having done so.
//!
//! Test commands can specify that a test should be rerun if one of the following (optional) is
//! specified and it matches the test's output:
//!
//! * `rerun-if-status` follows the same format as the `status`.
//! * `rerun-if-stderr` and `rerun-if-stdout` follow the same format as `stderr` and `stdout`.
//!
//! These can be useful if tests are subject to intermittent errors (e.g. network failure) that
//! should not be considered as a failure of the test itself. Test commands are rerun at most *n*
//! times, which by default is specified as 3. If no `rerun-if-` is specified, then the first time
//! a test fails, it will be reported to the user.
//!
//! The above file thus contains 4 meaningful tests, two specified by the user and two implied by
//! defaults: the `Compiler` should succeed (e.g. return a `0` exit code when run on Unix), and
//! its `stderr` output should warn about an unused variable on line 12; and the resulting binary
//! should succeed produce `Hello world` on `stdout`.
//!
//! A file's tests can be ignored entirely with:
//!
//! * `ignore-if: <cmd>` defines a shell command that will be run to determine whether to ignore
//! this test or not. If `<cmd>` returns 0 the test will be ignored, otherwise it will be run.
//! `<cmd>` will have its directory set to `CARGO_MANIFEST_DIR`.
//!
//! `lang_tester`'s output is deliberately similar to Rust's normal testing output. Running the
//! example `rust_lang_tester` in this crate produces the following output:
//!
//! ```text
//! $ cargo run --example=rust_lang_tester
//! Compiling lang_tester v0.1.0 (/home/ltratt/scratch/softdev/lang_tester)
//! Finished dev [unoptimized + debuginfo] target(s) in 3.49s
//! Running `target/debug/examples/rust_lang_tester`
//!
//! running 4 tests
//! test lang_tests::no_main ... ok
//! test lang_tests::unknown_var ... ok
//! test lang_tests::unused_var ... ok
//! test lang_tests::exit_code ... ok
//!
//! test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
//! ```
//!
//! If you want to run a subset of tests, you can specify simple filters which use substring match
//! to run a subset of tests:
//!
//! ```text
//! $ cargo run --example=rust_lang_tester var
//! Compiling lang_tester v0.1.0 (/home/ltratt/scratch/softdev/lang_tester)
//! Finished dev [unoptimized + debuginfo] target(s) in 3.37s
//! Running `target/debug/examples/rust_lang_tester var`
//!
//! running 2 tests
//! test lang_tests::unknown_var ... ok
//! test lang_tests::unused_var ... ok
//!
//! test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out
//! ```
//!
//! ## Integration with Cargo.
//!
//! Tests created with lang_tester can be used as part of an existing test suite and can be run
//! with the `cargo test` command. For example, if the Rust source file that runs your lang tests
//! is `lang_tests/run.rs` then add the following to your Cargo.toml:
//!
//! ```text
//! [[test]]
//! name = "lang_tests"
//! path = "lang_tests/run_tests.rs"
//! harness = false
//! ```
#![allow(clippy::needless_doctest_main)]
#![allow(clippy::new_without_default)]
#![allow(clippy::redundant_closure)]
#![allow(clippy::type_complexity)]
mod parser;
mod tester;
pub use tester::LangTester;
pub(crate) fn fatal(msg: &str) -> ! {
eprintln!("\nFatal exception:\n {}", msg);
std::process::exit(1);
}