layered-crate

Enforce dependencies amongst internal modules in a crate

0.2.0 -> 0.3.0, this tool is changed to a CLI tool rather than a proc-macro crate. See this issue for details

# install with cargo-binstall from binary release
cargo binstall layered-crate
# or build the tool from source
cargo install layered-crate

# check internal dependencies amongst layers, unused dependencies
# are automatically denied
layered-crate

CARGO=/my-cargo layered-crate -- +nightly check --lib --features ... 
# ^ change the cargo binary with env
#                                ^ pass extra args to cargo after --

The Problem

In a large Rust project, it's common to have modules or subsystems in a crate that depends on other parts of the crate, forming an internal dependency graph amongst modules. Since Rust allows you to import anything anywhere in the same crate, the dependency can become a mess over long time.

Some projects solve this using a workspace with multiple crates and use crate-level dependency. That's what happens when you see a bunch of project-* crates when searching for something on crates.io. There are several upsides and downsides to this. Just to list a few:

Upsides:
- Uses the standard Cargo.toml, which is more stable
- Might be better to split large code base, so someone doesn't have to download everything
- Might be better for incremental build, but I am clueless if this is true
Downsides:
- Need to publish 50 instead of 1 crate
- Need to have a more complicated Cargo.toml setup
- Cannot have pub(crate) visibility or impl for types from dependencies
- Might be worse for optimization since one of the factor for inlining is if the inlining is across a crate boundary. However, I have no clue what degree of effect this has

This tool uses a Layerfile.toml to specify the internal dependencies, and automatically checks that the dependencies are respected in the code as if they were separate crates. This allows you to keep the code in a single crate while enforcing the internal dependencies without having to split the crate manually.

It is designed to work out of the box with existing code base by adding the Layerfile.toml file. However, there are some limitations and edge cases, especially regarding macros, that you should read about below if you have regular or procedural macros in your code.

Usage

To split your crate into layers, this tool expects your entry point (e.g. src/lib.rs) to contain module definitions that correspond to the layers you want to create. For example:

// src/lib.rs
mod layer1 { // inline module
    pub fn foo() {
        // ...
    }
}
pub use layer1::foo; // re-exporting the function
pub mod layer2; // non-inline module at layer2.rs or layer2/mod.rs

/* ... */

Note that both private and public items in the module are checked,

Then, create a Layerfile.toml next to Cargo.toml with the following content:

[crate]
exclude = [] 
# ^ optional, list of modules to delete when checking layers
# note this is different from ignoring the layer/module
# to ignore something, just don't have a [layer.<name>] section for it

[layer.layer1] # for each module you want to check in lib.rs, create a table for it
#      ^ `layer1` corresponds to `mod layer1` in the code above
depends-on = ["layer2"] # list of layers that this layer depends on
impl = [] # any layer specified here will be checked together, see below for more details

[layer.layer2]
# ^ if the layer is at the bottom (doesn't depend on any other layer),
# you still need to create an empty table for it like this

Now, simply run layered-crate to check for violations - you will get an error if anything in layer2 imports from layer1!

By default, unused layers specified in depends-on will automatically be denied by setting RUSTFLAGS=-Dunused-imports. you can use the --no-rust-flags option to prevent this tool from touching RUSTFLAGS.

layered-crate --no-rust-flags

During the layer checking, the layer and its dependencies are split into different crates, so features that normally would work for you in the same-crate setup might not work as expected. Please read the limitations below

`pub(crate)` visibility and `impl` for types from dependencies

If one of your layers depends on an item that is pub(crate) in a layer below, or needs to implement a type for a layer below, you will get an error since the layer and its dependencies are split into different crates during layer checking.

To workaround this, add the impl property to the layer in Layerfile.toml:

[layer.layer1]
depends-on = ["layer2"]
impl = ["layer2"]       # <- add this

[layer.layer2]

When checking layer1, the tool will also put layer2 in the same test crate as layer1. However, the check is loosened in this case, since layer1 can also import from layer2's dependencies (i.e. transitive dependencies).

layer2 still cannot import from layer1 - you will get an error when checking layer2

Crate name in macro expansion

Macro expansion can give some nasty errors - especially procedural macros. If your crate uses macros (including procedural macros), please read this issue on GitHub before considering this tool.

Build Scripts

If build.rs is found in the working directory (i.e. next to Cargo.toml), it will be copied to the generated test packages. The build script might need some modification to work when checking the layers.

If the build script reads or writes files within the package's source tree (usually implemented by using CARGO_MANIFEST_DIR or CARGO_MANIFEST_PATH environment variable), they need to be changed using LAYERED_CRATE_ORIGINAL_ prefixed version to point to the original paths as if you are running the build script in the original location. Note that the build script should still generate the source code to the same location as usual, since the generated package will link to the original package instead of copying all the source.
```
// change this:
let manifest_dir = env!("CARGO_MANIFEST_DIR");
// to:
let manifest_dir = std::env::var("LAYERED_CRATE_ORIGINAL_MANIFEST_DIR")
    .unwrap_or(env!("CARGO_MANIFEST_DIR").to_string());
```

If the build script needs to be adjusted depending on which layer(s) are being built and which layer is being tested, you can use LAYERED_CRATE_DEPS_LAYERS and LAYERED_CRATE_TESTING_LAYER environment variables.

When building the full package initially (before testing any layer), LAYERED_CRATE_TESTING_LAYER will be empty and LAYERED_CRATE_DEPS_LAYERS will contain all layers.

let testing_layer = std::env::var("LAYERED_CRATE_TESTING_LAYER").unwrap_or_default();
let deps_layers: Vec<_> = std::env::var("LAYERED_CRATE_DEPS_LAYERS")
    .unwrap_or_default()
    .split(',').collect();
let is_running_layered_crate = !testing_layer.is_empty() || !deps_layers.is_empty();
if is_running_layered_crate && deps_layers.contains("foo") {
    // the dependencies of the current layer being tested contains the
    // "foo" layer
    build_for_foo_layer();
}
if testing_layer.as_str() == "foo" {
    // the current layer being tested is "foo"
}

If the build script is agnostic of the location of the package, then no change is needed. For example, if only generating files to target directory and using paths relative to target. The target directory will be the one for the test package, not the original package.

Other Limitations

Here are some more limitations of the tool other than the ones mentioned above:

Currently, we can only check library targets. For binary target, you have to declare a library target, then use that in your main.rs:

# these are the defaults so you can omit them
[lib]
name = "my_lib"      
path = "src/lib.rs"

[[bin]]
name = "my_bin"
path = "src/main.rs"

// src/main.rs
fn main() { my_lib::main_internal() }

We do not support modules produced by macros in the entry point, as we purely parse the entry point as syntax tree. Macros in other modules are fine.
The artifacts from running this tool are separated from the artifacts of building/checking your package normally with cargo. This means a CI pipeline that runs both cargo check/cargo clippy and layered-crate have duplicated checks and may incur additional cost. If this is an issue, consider:
- Adding the output directory of this tool to the cache of your pipeline.
- Use layered-crate instead of cargo to run those checks.
```
# Replace:
cargo clippy -D warnings -D clippy::todo ... # rest of clippy flags
# With:
layered-crate -- clippy -D warnings -D clippy::todo ... # rest of clippy flags
```
  Since this tool does not copy your source files except for the entry point (lib.rs), the diagnostic messages will still be accurate.
- Not using this tool and use multiple crates to organize your project.
- Use a global cache helper like sccache, which I have not used before, so I am not sure if it works with check commands

layered-crate 0.4.3