Cargo (as of 1.37) does not nicely handle the scenario where features of a
dependent crate of multiple other crates in the tree vary.
To illustrate the problem consider crates A, B and C arranged as follows:
* Crate A and B are both members of a Cargo virtual manifest
* Crate C provides two features, F1 and F2
* Crate A requests feature F1 of C, crate B requests F2 of C
When crate A and B are built together, `cargo` builds C with both feature F1 and
F2 enabled (the union of all enabled features). However when A or B are built
individually, `cargo` builds C with only feature F1 or F2 enabled.
Unfortunately in all these cases, `cargo` will build crate C in the same target
location and the outputs for C will be recreated every time the features for crate C
change.
From a clean workspace, building A individually first will cause C to be built
as expected. Then build B individually and C will be re-built because F2 was
enabled instead of F1. Now rebuild A and observe that C will be re-built again
because F1 was re-enabled.
In practice this problem is much less obvious as both A and B likely to not have
a direct dependency on C, indirectly causing rebuilds of numerous other crates as well.
The `solana-crate-features` offers a workaround to this "feature thrashing"
problem by explicitly declaring all "C-like crates" with the union of all features
that any other crate in the tree (either explicitly or implicitly) enable. All
crates in the Solana source tree should depend on `solana-crate-features`.
### Adding new dependent crates
When unnecessary `cargo` rebuilds are observed, the first step is to figure what
dependent crate is suffering from feature thrashing.
This information is not readily available from the stock `cargo` program, so use
the following steps to produce a custom `cargo` program that will output the
necessary feature information to stderr during a build:
```bash
$ git clone git@github.com:rust-lang/cargo.git -b rust-1.38.0
$ cd cargo
$ git apply 0001-Print-package-features.patch
$ cargo build
$ mv ~/.cargo/bin/cargo ~/.cargo/bin/cargo.org
$ cp ./target/debug/cargo ~/.cargo/bin/cargo
```
Rebuild with the custom `cargo` and search for indications of crates getting
built with different features (repeated runs of `./scripts/cargo-install-all.sh`
work great for this). When the problematic crate is identified, add it as a
dependency of `solana-crate-features` with the union of all observed enabled
features for that crate.
### Appendix
This command will enable some additional cargo log output that can help debug
dependency problems as well:
```bash
export CARGO_LOG=cargo::core::compiler::fingerprint=info
```