1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
//! Manage and build extra binaries for integration tests as regular Rust
//! crates.
//!
//! If you have integration tests for things that involve subprocess management,
//! inter-process communication, or platform tools, you might need to write some
//! extra "supporting" binaries of your own to help with these tests. For
//! example, if you want to test that your code does the right thing with the
//! exit status for a managed subprocess, you might want a supporting binary
//! that can be made to exit with a certain status code. If you're testing an
//! IPC exchange, you might want to test against a binary "mock" that sends some
//! scripted replies.
//!
//! And if you're already using Cargo to build and test, it would be nice to be
//! able to write those extra binaries in Rust, near to the crate you're
//! testing, as Cargo projects themselves. Then at least you'll know that your
//! test environments will already have the right toolchain installed.
//!
//! *To some extent this is already possible without using this crate at all!*
//! If you want an extra binary, you could put it under your `src/bin` or
//! `examples` directory and use it that way. But there are limitations to
//! what's currently possible under Cargo alone:
//!
//! - Crate binaries eg. under `src/bin`, or listed under `[[bin]]` in
//!   `Cargo.toml`, can be found via the environment variable
//!   [`CARGO_BIN_EXE_<name>`][cargo-env] when running tests. But they have to
//!   share dependencies with your entire crate! So whatever your supporting
//!   binaries depend on, your entire crate has to depend on as well. This is
//!   discussed in [Cargo issue #1982][cargo-1982]
//!
//!     [cargo-env]: https://doc.rust-lang.org/cargo/reference/environment-variables.html#environment-variables-cargo-sets-for-crates
//!     [cargo-1982]: https://github.com/rust-lang/cargo/issues/1982
//!
//! - Example binaries (under `examples/` or `[[example]]`) use
//!   `[dev-dependencies]` instead. But they have no equivalent environment
//!   variable, and might not be built by the time your test runs.
//!
//! - More philosophically: such binaries are not examples, nor are they real
//!   applications. They might not use any aspect of your crate whatsoever. They
//!   might deliberately malfunction. It might be confusing to end users to find
//!   these alongside your other examples. It might just not be the kind of
//!   organisation you want for your tests.
//!
//! - Organising supporting binaries as workspace crates requires publishing
//!   every one of those crates to [`crates.io`](https://crates.io) (or whatever
//!   registry you're using), even if they have no use whatsoever outside of
//!   your crate's integration tests.
//!
//! This crate provides a way to work around those constraints. It has a simple
//! interface for invoking Cargo to build extra binaries organised in a separate
//! directory under your crate.
//!
//! The first thing to note is that these extra binaries *aren't* binaries
//! listed in your actual project's manifest. So start by picking a directory
//! name and put them in there eg. this project uses `testbins`. **This is not
//! going to be a workspace.** Under this directory you will have these extra
//! binaries in their own Cargo packages.
//!
//! The structure should look something like this:
//!
//! ```none
//! ├── Cargo.toml        (your crate's manifest)
//! ├── src
//! │  └── lib.rs         (your crate's lib.rs)
//! ├── tests
//! │  └── tests.rs       (your crate's tests, which want to use the supporting
//! │                      binaries below)
//! │
//! └── testbins          (all the extra binary projects are under this
//!    │                   directory)
//!    ├── test-something (first extra binary)
//!    │  ├── Cargo.toml  (extra binary manifest, name = "test-something")
//!    │  └── src
//!    │     └── main.rs  (extra binary source)
//!    ├── test-whatever  (another extra binary, name = "test-whatever")
//!    │  ├── Cargo.toml
//!    │  └── src
//!    │     └── main.rs
//!     ...etc...
//! ```
//!
//! > ## Note
//! >
//! > It can be useful to put an empty `[workspace]` section in the `Cargo.toml`
//! > for these test binaries, so that Cargo knows not to [look in parent
//! > directories][cargo-10872].
//!
//!   [cargo-10872]: https://github.com/rust-lang/cargo/issues/10872
//!
//! With this setup, you can now call [`build_test_binary("test-something",
//! "testbins")`](crate::build_test_binary). See how:
//!
//! - `"test-something"` is the binary name you'd pass to Cargo *in the child
//!   project* eg. if you changed directory to the nested project, you'd run
//!   `cargo build --bin test-something`; it also has to be the name of the
//!   subdirectory this project is in
//! - `"testbins"` is the directory relative to your real project's manifest
//!   containing this test binary project (and maybe others); think of it like
//!   you'd think of the `examples` or `tests` directory
//!
//! If you need to set different profiles or features, or have more control over
//! the directory structure, there is also [a builder API](crate::TestBinary).
//! Also see [`build_test_binary_once!()`](crate::build_test_binary_once) for a
//! macro that lazily builds the binary and caches the path.
//!
//! Here's an example of how you might use this crate's API in a test, with a
//! binary named `does-build`:
//!
//! ```rust
//! # use test_binary::build_test_binary;
//!
//! let test_bin_path = build_test_binary("does-build", "testbins")
//!     .expect("error building test binary");
//!
//! let mut test_bin_subproc = std::process::Command::new(test_bin_path)
//!     .spawn()
//!     .expect("error running test binary");
//!
//! // Test behaviour of your program against the mock binary eg. send it
//! // something on stdin and assert what it prints on stdout, do some IPC,
//! // check for side effects.
//!
//! assert!(test_bin_subproc
//!     .wait()
//!     .expect("error waiting for test binary")
//!     .success());
//! ```
//!
//! The result returned by these functions contains the path of the built binary
//! as a [`std::ffi::OsString`], which can be passed to
//! [`std::process::Command`] or other crates that deal with subprocesses. The
//! path is not resolved to an absolute path by this crate, although it might be
//! one anyway. Since it is the path provided by Cargo after being invoked in
//! the current process' working directory, it will be valid as long as you do
//! not change the working directory between obtaining it and using it.

#![forbid(unsafe_code)]
#![warn(missing_docs, missing_debug_implementations)]
#![cfg_attr(docsrs, feature(doc_cfg))]

use std::{
    ffi::OsString,
    io::{BufReader, Read},
    path::{Path, PathBuf},
    process::{Command, Stdio},
};

// For the build_test_binary_once macro.
pub use once_cell;
pub use paste;

mod stream;

// Internal macros for OsString boilerplate.

macro_rules! vec_oss {
    ($($item:expr),* $(,)?) => {
        vec![
            $(::std::ffi::OsString::from($item),)+
        ]
    };
}

macro_rules! push_oss {
    ($args:expr, $item:expr) => {
        $args.push(::std::ffi::OsString::from($item))
    };
}

/// Builder constructor for a test binary.
///
/// Start with [`TestBinary::relative_to_parent(name,
/// manifest)`](TestBinary::relative_to_parent) where
/// - `name` is the name of the binary in the child project's manifest
/// - `manifest` is the path to the manifest file for the test binary, relative
///   to the directory that the containing project is in. It should probably end
///   in `Cargo.toml`.
///
/// Note that you can pass a path in a cross-platform way by using
/// [`PathBuf::from_iter()`][std::path::PathBuf::from_iter()]:
///
/// ```
/// # use std::path::PathBuf;
/// # use test_binary::TestBinary;
/// TestBinary::relative_to_parent(
///     "does-build",
///     &PathBuf::from_iter(["testbins", "does-build", "Cargo.toml"]),
/// );
/// ```
#[derive(Debug)]
pub struct TestBinary<'a> {
    binary: &'a str,
    manifest: &'a Path,
    features: Vec<&'a str>,
    default_features: bool,
    profile: Option<&'a str>,
}

impl<'a> TestBinary<'a> {
    /// Creates a new `TestBinary` by specifying the child binary's manifest
    /// relative to the parent.
    pub fn relative_to_parent(name: &'a str, manifest: &'a Path) -> Self {
        Self {
            binary: name,
            manifest,
            features: vec![],
            default_features: true,
            profile: None,
        }
    }

    /// Specifies a profile to build the test binary with.
    pub fn with_profile(&mut self, profile: &'a str) -> &mut Self {
        self.profile = Some(profile);
        self
    }

    /// Specifies not to enable default features.
    pub fn no_default_features(&mut self) -> &mut Self {
        self.default_features = false;
        self
    }

    /// Specifies a feature to enable for the test binary. These are additive,
    /// so if you call this multiple times all the features you specify will be
    /// enabled.
    pub fn with_feature(&mut self, feature: &'a str) -> &mut Self {
        self.features.push(feature);
        self
    }

    /// Builds the binary crate we've prepared. This goes through Cargo, so it
    /// should function identically to `cargo build --bin testbin` along with
    /// any additional flags from the builder methods.
    pub fn build(&mut self) -> Result<OsString, TestBinaryError> {
        fn get_cargo_env(key: &str) -> Result<OsString, TestBinaryError> {
            std::env::var_os(key).ok_or_else(|| {
                TestBinaryError::NonCargoRun(format!(
                    "{} '{}' {}",
                    "The environment variable ", key, "is not set",
                ))
            })
        }

        let cargo_path = get_cargo_env("CARGO")?;

        // Resolve test binary project manifest.
        let mut manifest_path = PathBuf::from(get_cargo_env("CARGO_MANIFEST_DIR")?);
        manifest_path.push(self.manifest);

        let mut cargo_args = vec_oss![
            "build",
            "--message-format=json",
            "-q",
            "--manifest-path",
            manifest_path,
            "--bin",
            self.binary,
        ];

        if let Some(prof) = self.profile {
            push_oss!(cargo_args, "--profile");
            push_oss!(cargo_args, prof);
        }

        if !self.default_features {
            push_oss!(cargo_args, "--no-default-features");
        }

        for feature in &self.features {
            push_oss!(cargo_args, "--features");
            push_oss!(cargo_args, feature);
        }

        let mut cargo_command = Command::new(cargo_path)
            .args(cargo_args)
            .stdout(Stdio::piped())
            .stderr(Stdio::piped())
            .spawn()?;

        let reader = BufReader::new(
            // The child process' stdout being None is legitimately a
            // programming error, since we created it ourselves two lines ago.
            //
            // Use as_mut() instead of take() here because if we detach
            // ownership from the subprocess, we risk letting it drop
            // prematurely, which can make it close before the subprocess is
            // finished, resulting in a broken pipe error (but in a highly
            // timing/platform/performance dependent and intermittent way).
            cargo_command
                .stdout
                .as_mut()
                .expect("Cargo subprocess output has already been claimed"),
        );

        let cargo_outcome = stream::process_messages(reader, self.binary);

        // See above re. stderr being None.
        let mut error_reader = BufReader::new(
            cargo_command
                .stderr
                .as_mut()
                .expect("Cargo subprocess error output has already been claimed"),
        );

        let mut error_msg = String::new();
        error_reader.read_to_string(&mut error_msg)?;

        if cargo_command.wait()?.success() {
            // The process succeeded. There should be a result from the JSON
            // output above.
            cargo_outcome
                .expect("Cargo succeeded but produced no output")
                .map(Into::into)
        } else if let Some(Err(err)) = cargo_outcome {
            // The process failed and there's an error we extracted from the
            // JSON output. Usually this means a compiler error.
            Err(err)
        } else {
            // The process failed but there's no error from the JSON output.
            // This will happen if there's an invocation error eg. the manifest
            // does not exist.
            //
            // This case also covers process failure but an Ok() result from the
            // above message parsing. This would be strange (if it's even
            // possible), but if it happens we should still report the error.
            Err(TestBinaryError::CargoFailure(error_msg))
        }
    }
}

/// Simplified function for building a test binary where the binary is in a
/// subdirectory of the same name, the manifest is named `Cargo.toml`, and you
/// don't need any non-default features or to specify a profile.
///
/// For example, if your parent contains the child binary in
/// `testbins/does-build`, and the binary is named `does-build` in its
/// `Cargo.toml`, then you can just call `build_test_binary("does_build",
/// "testbins")`.
pub fn build_test_binary<R: AsRef<Path>>(
    name: &str,
    directory: R,
) -> Result<OsString, TestBinaryError> {
    TestBinary::relative_to_parent(
        name,
        &PathBuf::from_iter([directory.as_ref(), name.as_ref(), "Cargo.toml".as_ref()]),
    )
    .build()
}

/// Error type for build result.
#[derive(thiserror::Error, Debug)]
pub enum TestBinaryError {
    /// We are not running under Cargo.
    #[error("{0}; is this running under a 'cargo test' command?")]
    NonCargoRun(String),
    /// An error running Cargo itself.
    #[error("IO error running Cargo")]
    CargoRunError(#[from] std::io::Error),
    /// Cargo ran but did not succeed.
    #[error("Cargo failed, stderr: {0}")]
    CargoFailure(String),
    /// Cargo ran but there was a compilation error.
    #[error("build error:\n{0}")]
    BuildError(String),
    /// Cargo ran and seemed to succeed but the requested binary did not appear
    /// in its build output.
    #[error(r#"could not find binary "{0}" in Cargo output"#)]
    BinaryNotBuilt(String),
}

/// Generate a singleton function to save invoking Cargo multiple times for the
/// same binary.
///
/// This is useful when you have many integration tests that use the one test
/// binary, and don't want to invoke Cargo over and over for each one. Note that
/// Cargo itself implements both locking and caching at the filesystem level, so
/// all this macro will save you is the overhead of spawning the Cargo process
/// to do its checks. That may still be appreciable for high numbers of tests or
/// on slow systems.
///
/// Calling `build_test_binary_once!(binary_name, "tests_dir")` (no quotes on
/// `binary_name`) will generate a function `path_to_binary_name()` that returns
/// the path of the built test binary as an `OsString`, just like
/// `build_test_binary("binary_name", "tests_dir")` would. Unlike
/// `build_test_binary()`, the generated function will only build the binary
/// once, and only on the first call. Subsequent calls will use a cached path
/// and assume the initial build is still valid. The generated function unwraps
/// the result internally and will panic on build errors.
///
/// For example, if you use `build_test_binary_once!(my_test, "testbins")` in
/// `tests/common/mod.rs`, that module will then contain a function
/// `path_to_my_test() -> std::ffi::OsString`. Multiple integration tests can
/// then use `common::path_to_my_test()` to obtain the path. Cargo will only be
/// run once for this binary, even if the integration tests that use it are
/// being run in multiple threads.
///
/// > ## Note
/// >
/// > That this means the binary name must be a valid identifier eg. not have
/// > dashes in it.
///
/// ```rust
/// # use test_binary::build_test_binary_once;
/// // Build a test binary named "multiple".
/// build_test_binary_once!(multiple, "testbins");
///
/// // The first test that gets run will cause the binary "multiple" to be built
/// // and the path will be cached inside the `path_to_multiple()` function.
///
/// let test_bin_path = path_to_multiple();
/// assert!(std::process::Command::new(test_bin_path)
///     .status()
///     .expect("Error running test binary")
///     .success());
///
/// // Subsequent tests will just get the cached path without spawning Cargo
/// // again.
///
/// let test_bin_path_again = path_to_multiple();
/// assert!(std::process::Command::new(test_bin_path_again)
///     .status()
///     .expect("Error running test binary")
///     .success());
/// ```
///
/// If you need to use extra features or a non-default profile, you will need to
/// go back to using the builder.
#[macro_export]
macro_rules! build_test_binary_once {
    ($name:ident, $tests_dir:expr) => {
        $crate::paste::paste! {
            pub fn [<path_to_ $name>]() -> std::ffi::OsString {
                use $crate::once_cell::sync::Lazy;
                use std::ffi::OsString;

                static [<LAZY_PATH_TO_ $name>]: Lazy<OsString> =
                    Lazy::new(|| $crate::build_test_binary(
                        stringify!($name),
                        $tests_dir
                    ).unwrap());
                [<LAZY_PATH_TO_ $name>].clone()
            }
        }
    };
}