1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184

// Copyright (c) The cargo-guppy Contributors
// SPDX-License-Identifier: MIT OR Apache-2.0

//! About workspace-hack crates, how `cargo hakari` manages them, and how much faster they make builds.
//!
//! # What are workspace-hack crates?
//!
//! Let's say you have a Rust crate `my-crate` with two dependencies:
//!
//! ```toml
//! # my-crate/Cargo.toml
//! [dependencies]
//! foo = "1.0"
//! bar = "2.0"
//! ```
//!
//! Let's say that `foo` and `bar` both depend on `baz`:
//!
//! ```toml
//! # foo-1.0/Cargo.toml
//! [dependencies]
//! baz = { version = "1", features = ["a", "b"] }
//!
//! # bar-2.0/Cargo.toml
//! [dependencies]
//! baz = { version = "1", features = ["b", "c"] }
//! ```
//!
//! What features is `baz` built with?
//!
//! One way to resolve this question might be to build `baz` twice with each requested set of
//! features. But this is likely to cause a combinatorial explosion of crates to build, so Cargo
//! doesn't do that. Instead,
//! [Cargo builds `baz` once](https://doc.rust-lang.org/nightly/cargo/reference/features.html?highlight=feature#feature-unification)
//! with the *union* of the features enabled for the package: `[a, b, c]`.
//!
//! ---
//!
//! **NOTE:** This description elides some details around unifying build and dev-dependencies: for
//! more about this, see the documentation for guppy's
//! [`CargoResolverVersion`](guppy::graph::cargo::CargoResolverVersion).
//!
//! ---
//!
//! Now let's say you're in a workspace, with a second crate `your-crate`:
//!
//! ```toml
//! # your-crate/Cargo.toml
//! [dependencies]
//! baz = { version = "1", features = ["c", "d"] }
//! ```
//!
//! In this situation:
//!
//! | if you build                                 | `baz` is built with |
//! | -------------------------------------------- | ------------------- |
//! | just `my-crate`                              | `a, b, c`           |
//! | just `your-crate`                            | `c, d`              |
//! | `my-crate` and `your-crate` at the same time | `a, b, c, d`        |
//!
//! Even in this simplified scenario, there are three separate ways to build `baz`. For a dependency
//! like [`syn`](https://crates.io/crates/syn) that has
//! [many optional features](https://github.com/dtolnay/syn#optional-features),
//! large workspaces end up with a very large number of possible build configurations.
//!
//! Even worse, the feature set of a package affects everything that depends on it, so `syn`
//! being built with a slightly different feature set than before would cause *every package that
//! directly or transitively depends on `syn` to be rebuilt. For large workspaces, this can result
//! a lot of wasted build time.
//!
//! ---
//!
//! To avoid this problem, many large workspaces contain a `workspace-hack` crate. The
//! purpose of this package is to ensure that dependencies like `syn` are always built with the same
//! feature set no matter which workspace packages are currently being built. This is done by:
//! 1. adding dependencies like `syn` to `workspace-hack` with the full feature set required by any
//!   package in the workspace
//! 2. adding `workspace-hack` as a dependency of every crate in the repository.
//!
//! Some examples of `workspace-hack` packages:
//!
//! * Rust's [`rustc-workspace-hack`](https://github.com/rust-lang/rust/blob/0bfc45aa859b94cedeffcbd949f9aaad9f3ac8d8/src/tools/rustc-workspace-hack/Cargo.toml)
//! * Firefox's [`mozilla-central-workspace-hack`](https://hg.mozilla.org/mozilla-central/file/cf6956a5ec8e21896736f96237b1476c9d0aaf45/build/workspace-hack/Cargo.toml)
//! * Diem's [`diem-workspace-hack`](https://github.com/diem/diem/blob/91578fec8d575294b47b3ee7af691fd9dc6eb240/common/workspace-hack/Cargo.toml)
//!
//! These packages have historically been maintained by hand, on a best-effort basis.
//!
//! # What can hakari do?
//!
//! Maintaining workspace-hack packages manually can result in:
//! * Missing crates
//! * Missing feature lists for crates
//! * Outdated feature lists for crates
//!
//! All of these can result in longer than optimal build times.
//!
//! `cargo hakari` can automate the maintenance of these packages, greatly reducing the amount of
//! time and effort it takes to maintain these packages.
//!
//! # How does hakari work?
//!
//! `cargo hakari` uses [guppy]'s Cargo build simulations to determine the full set of features
//! that can be built for each package. It then looks for
//!
//! For more details about the algorithm, see the documentation for the [`hakari`] library.
//!
//! # How much faster do builds get?
//!
//! The amount to which builds get faster depends on the size of the repository. In general, the
//! benefit grows super-linearly with the size of the workspace and the number of crates in it.
//!
//! On moderately large workspaces with several hundred third-party dependencies, a cumulative
//! performance benefit of 20-25% has been seen. Individual commands can be anywhere from 10%
//! to 95+% faster. `cargo check` often benefits more than `cargo build` because expensive
//! linker invocations aren't a factor.
//!
//! ## Performance metrics
//!
//! All measurements were taken on the following system:
//!
//! * **Processor:** AMD Ryzen 9 3900X processor (12 cores, 24 threads)
//! * **Memory:** 64GB
//! * **Operating system:** [Pop!_OS 21.04](https://pop.system76.com/), running Linux kernel 5.13
//! * **Filesystem:** btrfs
//!
//! ---
//!
//! On the [Diem repository](https://github.com/diem/diem/), at revision 6fa1c8c0, with the following
//! `cargo build` commands in sequence:
//!
//! | Command                               | Before (s) | After (s) | Change   | Notes                                        |
//! |---------------------------------------|-----------:|----------:|---------:|----------------------------------------------|
//! | `-p move-lang`                        | 35.56      | 53.06     | 49.21%   | First command has to build more dependencies |
//! | `-p move-lang --all-targets`          | 46.64      | 25.45     | -45.44%  |                                              |
//! | `-p move-vm-types`                    | 10.56      | 0.29      | -97.24%  | This didn't have to build anything           |
//! | `-p network`                          | 19.16      | 14.10     | -26.42%  |                                              |
//! | `-p network --all-features`           | 21.59      | 18.20     | -15.70%  |                                              |
//! | `-p storage-interface`                | 7.04       | 2.97      | -57.83%  |                                              |
//! | `-p storage-interface --all-features` | 12.78      | 1.15      | -91.03%  |                                              |
//! | `-p diem-node`                        | 102.32     | 84.65     | -17.27%  | This command built a large C++ dependency    |
//! | `-p backup-cli`                       | 52.47      | 33.26     | -36.61%  | Linked several binaries                      |
//! | **Total**                             | 308.12     | 233.12    | -24.34%  |                                              |
//!
//! With the following `cargo check` commands in sequence:
//!
//! | Command                               | Before (s) | After (s) | Change  | Notes                                         |
//! |---------------------------------------|-----------:|----------:|--------:|-----------------------------------------------|
//! | `-p move-lang`                        | 16.04      | 36.55     | 127.83% | First command has to build more dependencies  |
//! | `-p move-lang --all-targets`          | 26.73      | 13.22     | -50.56% |                                               |
//! | `-p move-vm-types`                    | 9.41       | 0.29      | -96.91% | This didn't have to build anything            |
//! | `-p network`                          | 12.41      | 9.43      | -24.01% |                                               |
//! | `-p network --all-features`           | 15.12      | 11.54     | -23.69% |                                               |
//! | `-p storage-interface`                | 5.33       | 1.65      | -68.98% |                                               |
//! | `-p storage-interface --all-features` | 8.22       | 1.02      | -87.59% |                                               |
//! | `-p diem-node`                        | 56.60      | 51.29     | -9.38%  | This command built two large C++ dependencies |
//! | `-p backup-cli`                       | 13.57      | 5.51      | -59.40% |                                               |
//! | **Total**                             | 163.44     | 130.50    | -20.15% |                                               |//!
//!
//! On the much smaller [cargo-guppy repository](https://github.com/facebookincubator/cargo-guppy),
//! at revision 65e8c8d7, with the following `cargo build` commands in sequence:
//!
//! | Command                    | Before (s) | After (s) | Change  | Notes                                        |
//! |----------------------------|-----------:|----------:|--------:|----------------------------------------------|
//! | `-p guppy`                 | 11.77      | 13.48     | 14.53%  | First command has to build more dependencies |
//! | `-p guppy --all-features`  | 9.83       | 9.72      | -1.12%  |                                              |
//! | `-p hakari`                | 6.03       | 3.75      | -37.94% |                                              |
//! | `-p hakari --all-features` | 10.78      | 10.28     | -4.68%  |                                              |
//! | `-p determinator`          | 4.60       | 3.90      | -15.22% |                                              |
//! | `-p cargo-hakari`          | 17.72      | 7.22      | -59.26% |                                              |
//! | **Total**                  | 60.73      | 48.34     | -20.41% |                                              |
//!
//! # Drawbacks
//!
//! * The first build in a workspace will take longer because more dependencies have to be cached.
//!   - This also applies to builds performed after `cargo clean`, or after Rust version upgrades.
//!   - However, this usually pays off over time.
//! * Some crates may accidentally start skipping features they really need, because the
//!   workspace-hack turns those features on for them.
//!   - This is not a major issue for repositories that don't release crates to `crates.io`.
//!   - It can also be caught at publish time, or with a periodic CI job that does a build after
//!     running `cargo hakari disable`.
//! * Publishing becomes more complex as the workspace-hack dependency needs to be removed from
//!   `Cargo.toml`.
//!   - `cargo hakari` provides a `publish` command to automatically do this for you.