<div align="center">
<img src="https://raw.githubusercontent.com/tracel-ai/xtask/update_readme/assets/xtask.png" width="256px"/>
<h1>Tracel Xtask</h1>
[](https://discord.gg/uPEBbYYDB6)
[](https://crates.io/crates/tracel-xtask)
[](https://crates.io/crates/tracel-xtask)
[](https://github.com/tracel-ai/xtask/actions/workflows/ci.yml)
---
<br/>
</div>
A collection of easy to use and extend commands to be used in your [xtask CLI][1] based on [clap][2].
These commands are used in our Tracel repositories. By centralizing all of our redundant commands we can save a big amount
of code duplication, boilerplate and considerably lower the burden maintenance. It also provides a unified interface across
all of our repositories.
These commands are not specific to Tracel repositories and they should be pretty much usable in any Rust repositories with
a cargo workspace as well as other repositories where Rust is not necessarily the only language. The commands can be easily
extended using handy proc macros and by following some patterns described in this README.
## Getting Started
### Setting Up a Cargo Workspace with an xtask binary crate
1. **Create a new Cargo workspace:**
```bash
cargo new my_workspace --vcs none
cd my_workspace
```
2. **Create the `xtask` binary crate:**
```bash
cargo new xtask --bin --vcs none
```
3. **Configure the workspace:**
Edit the `Cargo.toml` file in the root of the workspace to include the following:
```toml
[workspace]
members = ["xtask"]
```
4. **Add the `tracel-xtask` dependency:**
In the `xtask/Cargo.toml` file, add the following under `[dependencies]`:
```toml
[dependencies]
tracel-xtask = "1.0.0"
```
5. **Build the workspace:**
```bash
cargo build
```
Your workspace is now set up with a `xtask` binary crate that depends on `tracel-xtask` version 1.0.0.
### Bootstrap main.rs
1. In the `main.rs` file of the newly created `xtask` crate, import the `tracel_xtask prelude` and then declare
a `Command` enum along with the `macros::base_commands` attribute to select the base commands you want to use from
the collection:
```rust
use tracel_xtask::prelude::*;
#[macros::base_commands(
Bump,
Check,
Fix
Test,
)]
pub enum Command {}
```
2. Update the `main` function to initialize the xtask parser and dispatch the base commands:
```rust
fn main() -> anyhow::Result<()> {
let args = init_xtask::<Command>()?;
match args.command {
// dispatch_base_commands function is generated by the commands macro
_ => dispatch_base_commands(args),
}
}
```
3. Build the workspace with `cargo build` at the root of the repository to verify everything is fine before
proceeding to the next section.
3. You should now be able to display the main help screen which lists the commands you selected previously:
```sh
cargo xtask --help
```
### Setup aliases for easy invocation
#### Cargo Alias
Invoking the xtask binary with cargo is very verbose and not really usable as is. Happily we can create a
cargo alias to make it really effortless to invoke it.
Create a new file `.cargo/config.toml` in your repository with the following contents:
```toml
[alias]
xtask = "run --target-dir target/xtask --package xtask --bin xtask --"
```
This saves quite a few characters to type :-) You can now invoke `xtask` like this at the root of your
repository:
```sh
cargo xtask
```
Try it with `cargo xtask --help`.
#### Shell Alias
We can save even more typing by creating a shell alias for `cargo xtask`.
For instance we can set the alias to `cx`. Here is how to do it in various shells.
- For bash:
```bash
nano ~/.bashrc
# add this to the file
alias cx='cargo xtask'
# save and source the file or restart the shell session
source ~/.bashrc
```
- For zsh:
```sh
nano ~/.zshrc
# add this to the file
alias cx='cargo xtask'
# save and source the file or restart the shell session
source ~/.zshrc
```
- For fish:
```fish
nano ~/.config/fish/config.fish
# add this to the file
alias cx='cargo xtask'
# save and source the file or restart the shell session
source ~/.config/fish/config.fish
```
- For powershell:
```powershell
notepad $PROFILE
# add this at the end of file
function cx {
cargo xtask $args
}
# save and quit then open a new powershell terminal
```
Try it with `cx --help` at the root of the repository.
## Conventions
### Repository structure
All our repositories follow the same directory hierarchy:
- a `crates` directory which contains all the crates of the workspace
- an `examples` directory which holds all the examples crates
- a `xtask` directory which is binary crate of the CLI using `xtask-common`
### Integration tests
[Integration tests][3] are tests usually contained in a `tests` directory. `tracel-xtask` test command expects all
the integration file names to be prefixed with `test_`. If a file in the `tests` directory of a crate does not start
with `test_` then it will be ignored when running the `cargo xtask test integration` command.
## Interface generalities
### Target
There are 4 default targets provided by the tracel-xtask command line interface:
- `workspace` which targets the whole workspace, this is the default target
- `crates` are all the binary crates and library crates
- `examples` are all the example crates
- `all-packages` are both `crates` and `examples` targets
`workspace` and `all-packages` are different because `workspace` uses the `--workspace` flag of cargo whereas `all-packages`
relies on `crates` and `examples` targets which use the `--package` flag.
Here are some examples:
```sh
# run all the crates tests
cargo xtask test --target crates all
# check format for examples, binaries and libs
cargo xtask check --target all-packages unit
# build the workspace
cargo xtask build --target workspace
# workspace is the default target so this has the same effect
cargo xtask build
```
### Global options
The following options are global and must be used before the actual command on the command line, for instance:
- Environment (`-e`, `--environment`):
```sh
cargo xtask -e production build
```
`-e` or `--environment` does not do anything per se in the base commands, it is a flag whose only goal is
to inform your custom commands or dispatch functions about the targeted environment which can be `development` (default),
`staging` or `production`.
- Execution environment (`-E`, `--execution-environment`):
```sh
cargo xtask -E no-std build
```
`-E` or `--execution-environment` does not do anything per se in the base commands, it is a flag whose only goal is
to inform your custom commands or dispatch functions about the targeted execution environment which can be `std` or
`no-std`.
- Coverage (`-c`, `--enable-coverage`):
`-c` or `--enabled-coverage` which setup Rust toolchain to generate coverage information.
## Anatomy of a base command
We use the derive API of clap which is based on struct, enum and attribute proc macros. Each base command is a
submodule of the `base_commands` module. If the command has arguments there is a struct named `<command>CmdArgs` which
declare the options, arguments and subcommands. In the case of subcommands a corresponding enum named `<command>SubCommand`
is defined.
Here is an example for a `foo` command:
```rust
#[macros::declare_command_args(Target, FooSubCommand)]
struct FooCmdArgs {}
pub enum FooSubCommand {
/// A sub command for foo (usage on the command line: cargo xtask foo print-something)
PrintSomething,
}
```
Note that it is possible to have an arbitrary level of nested subcommands but deeper nested subcommands cannot be extended,
in other words, only the first level of subcommands can be extended. If possible, try to design commands with only one
level of subcommands to keep the interface simple.
In the following sections we will see how to create completely new commands as well how to extend existing bae commands.
## Customization
### Create a new command
1. First, we organize commands by creating a `commands` module. Create a file `xtask/src/commands/mycommand.rs` as well
as the corresponding `mod.rs` file to declare the module contents.
2. Then, in `mycommand.rs` define the arguments struct with the `declare_command_args` macro and the handle_command function.
The `declare_command_args` takes two parameters, the first is the type of the target enum and the second is the type of
the subcommand enum if any. If the command has no target r no subcommand then put `None` for each argument respectively.
`Target` is the default target type provided by `tracel-xtask` as it is described in the Generalities section. You can
create your type from scratch or even extend `Target` as we will see in a later section.
```rust
use tracel_xtask::prelude::*;
#[macros::declare_command_args(Target, None)]
struct MyCommandCmdArgs {}
pub fn handle_command(_args: MyCommandCmdArgs) -> anyhow::Result<()> {
println!("Hello from my-command");
Ok(())
}
```
3. Make sure to update the `mod.rs` file to declare the command module:
```
pub(crate) mod my_command;
```
4. We can now add a new variant to the `Command` enum in `main.rs`:
```rust
mod commands;
use tracel_xtask::prelude::*;
#[macros::base_commands(
Bump,
Check,
Fix,
Test,
)]
pub enum Command {
MyCommand(commands::mycommand::MyCommandCmdArgs),
}
```
5. And dispatch to our command implementation module:
```rust
fn main() -> anyhow::Result<()> {
let args = init_xtask::<Command>()?;
match args.command {
Command::NewCommand(args) => commands::new_command::handle_command(args),
_ => dispatch_base_commands(args),
}
}
```
6. You can now test your new command with:
```sh
cargo xtask my-command --help
cargo xtask my-command
```
### Extend the default Target enum
Let's implement a new commands called `extended-target` to illustrate how to extend the default Target enum.
1. Create the `commands/extended_target.rs` file and update the `mod.rs` file as we saw in the previous section.
2. We also need to add a new `strum` dependency to our `Cargo.toml` file:
```toml
[dependencies]
strum = {version = "0.26.3", features = ["derive"]}
```
3. Then we can extend the `Target` enum with the `macros::extend_targets` attribute in our `extended_target.rs` file.
Here we choose to add a new target called `frontend` which targets a frontend we could find in a monorepo:
```rust
use tracel_xtask::prelude::*;
#[macros::extend_targets]
pub enum MyTarget {
/// Target the frontend component of the monorepo.
Frontend,
}
```
4. Then we define our command arguments by referencing our newly created `MyTarget` enum:
```rust
#[macros::declare_command_args(MyTarget, None)]
struct ExtendedTargetCmdArgs {}
```
5. Our new target is then available for use in the `handle_command` function:
```rust
pub fn handle_command(args: ExtendedTargetCmdArgs) -> anyhow::Result<()> {
match args.target {
// Default targets
MyTarget::AllPackages => println!("You chose the target: all-packages"),
MyTarget::Crates => println!("You chose the target: crates"),
MyTarget::Examples => println!("You chose the target: examples"),
MyTarget::Workspace => println!("You chose the target: workspace"),
// Additional target
MyTarget::Frontend => println!("You chose the target: frontend"),
};
Ok(())
}
```
6. Register our new command the usual way by adding it to our `Command` enum and dispatch it
in the `main` function:
```rust
mod commands;
use tracel_xtask::prelude::*;
#[macros::base_commands(
Bump,
Check,
Fix,
Test,
)]
pub enum Command {
ExtendedTarget(commands::extended_target::ExtendedTargetCmdArgs),
}
fn main() -> anyhow::Result<()> {
let args = init_xtask::<Command>()?;
match args.command {
Command::ExtendedTarget(args) => commands::extended_target::handle_command(args),
_ => dispatch_base_commands(args),
}
}
```
7. Test the command with:
```rust
cargo xtask extended-target --help
cargo xtask extended-target --target frontend
```
### Extend a base command
To extend an existing command we use the `macros::extend_command_args` attribute which takes three parameters:
- first argument is the type of the base command arguments struct to extend,
- second argument is the target type (or `None` if there is no target),
- third argument is the subcommand type (or `None` if there is no subcommand).
Let's use two examples to illustrate this, the first is a commands that extends the `build` base commande with
a new `--debug` argument, and the second is a new command that extends the subcommands of the `check` base command
to add a new `my-check` subcommand.
You can find more examples in the `xtask` crate of this repository.
#### Extend the arguments of a base command
We create a new command called `extended-build-args` which has an additional arguments `--debug`.
1. Create the `commands/extended_build_args.rs` file and update the `mod.rs` file as we saw in the previous section.
2. Extend the `BuildCommandArgs` struct using macro `macros::extend_command_args` and define the `handle_command` function.
Note that the macro automatically implements the `TryInto` trait which makes it easy to dispatch back to the base command
own `handle_command` function. Also note that if the base command requires a target they you need to provide one as well
in your extension, i.e. the target parameter of the macro cannot be `None` if the base command has a `Target`.
```rust
use tracel_xtask::prelude::*;
#[macros::extend_command_args(BuildCmdArgs, Target, None)]
pub struct ExtendedBuildArgsCmdArgs {
/// Print additional debug info when set
#[arg(short, long)]
pub debug: bool,
}
pub fn handle_command(args: ExtendedBuildArgsCmdArgs) -> anyhow::Result<()> {
if args.debug {
println!("Debug is enabled");
}
base_commands::build::handle_command(args.try_into().unwrap())
}
```
3. Register the new command the usual way by adding it to our `Command` enum and dispatch it
in the `main` function:
```rust
mod commands;
use tracel_xtask::prelude::*;
#[macros::base_commands(
Bump,
Check,
Fix,
Test,
)]
pub enum Command {
ExtendedBuildArgs(commands::extended_build_args::ExtendedBuildArgsCmdArgs),
}
fn main() -> anyhow::Result<()> {
let args = init_xtask::<Command>()?;
match args.command {
Command::ExtendedBuildArgs(args) => commands::extended_build_args::handle_command(args),
_ => dispatch_base_commands(args),
}
}
```
4. Test the command with:
```rust
cargo xtask extended-build-args --help
cargo xtask extended-build-args --debug
```
#### Extend the subcommands of a base command
For this one we create a new command called `extended-check-subcommands`.
1. Create the `commands/extended_check_subcommands.rs` file and update the `mod.rs` file as we saw in the previous section.
2. Extend the `CheckCommandArgs` struct using macro `macros::extend_command_args`:
```rust
use tracel_xtask::prelude::*;
#[macros::extend_command_args(CheckCmdArgs, Target, ExtendedCheckSubcommand)]
pub struct ExtendedCheckedArgsCmdArgs {}
```
3. Implement the `ExtendedCheckSubcommand` enum by extending the `CheckSubcommand` base enum with the macro `extend_subcommands`.
It takes the name of the type of the subcommand enum to extend:
```rust
#[macros::extend_subcommands(CheckSubCommand)]
pub enum ExtendedCheckSubcommand {
/// An additional subcommand for our extended check command.
MySubcommand,
}
```
4. Implement the `handle_command` function to handle the new subcommand. Note that we also must handle the `All` variant:
```rust
use strum::IntoEnumIterator;
pub fn handle_command(args: ExtendedCheckedArgsCmdArgs) -> anyhow::Result<()> {
match args.get_command() {
ExtendedCheckSubcommand::MySubcommand => run_my_subcommand(args.clone()),
ExtendedCheckSubcommand::All => {
ExtendedCheckSubcommand::iter()
.filter(|c| *c != ExtendedCheckSubcommand::All)
.try_for_each(|c| {
handle_command(
ExtendedCheckedArgsCmdArgs {
command: Some(c),
target: args.target.clone(),
exclude: args.exclude.clone(),
only: args.only.clone(),
},
)
})
}
_ => base_commands::check::handle_command(args.try_into().unwrap()),
}
}
fn run_my_subcommand(_args: ExtendedCheckedArgsCmdArgs) -> Result<(), anyhow::Error> {
println!("Executing new subcommand");
Ok(())
}
```
5. Register the new command the usual way by adding it to our `Command` enum and dispatch it
in the `main` function:
```rust
mod commands;
use tracel_xtask::prelude::*;
#[macros::base_commands(
Bump,
Check,
Fix,
Test,
)]
pub enum Command {
ExtendedCheckSubcommand(commands::extended_check_subcommands::ExtendedCheckedArgsCmdArgs),
}
fn main() -> anyhow::Result<()> {
let args = init_xtask::<Command>()?;
match args.command {
Command::ExtendedCheckSubcommand(args) => commands::extended_check_subcommands::handle_command(args),
_ => dispatch_base_commands(args),
}
}
```
6. Test the command with:
```rust
cargo xtask extended-check-subcommands --help
cargo xtask extended-check-subcommands my-check
```
## Custom builds and tests
`xtask-common` provides helper functions to easily execute custom builds or tests with specific features or build target (do not confuse
Rust build targets which is an argument of the `cargo build` command with the xtask target we introduced here).
For instance we can extend the `build` command using the `Extend an existing xtask-common command` section and use the helper function to
build additional crates with custom features or build targets:
```rust
pub fn handle_command(mut args: tracel_xtask::commands::build::BuildCmdArgs) -> anyhow::Result<()> {
// regular execution of the build command
tracel_xtask::commands::build::handle_command(args)?;
// additional crate builds
// build 'my-crate' with all the features
tracel_xtask::utils::helpers::custom_crates_build(vec!["my-crate"], vec!["--all-features"], None, None, "all features")?;
// build 'my-crate' with specific features
tracel_xtask::utils::helpers::custom_crates_build(vec!["my-crate"], vec!["--features", "myfeature1,myfeature2"], None, None, "myfeature1,myfeature2")?;
// build 'my-crate' with a different target than the default one
tracel_xtask::utils::helpers::custom_crates_build(vec!["my-crate"], vec!["--target", "thumbv7m-none-eabi"], None, None, "thumbv7m-none-eabi target")?;
Ok(())
}
```
## Enable and generate coverage information
Here is a example GitHub job which shows how to setup coverage, enable it and upload coverage information to codecov:
```yaml
env:
GRCOV_LINK: "https://github.com/mozilla/grcov/releases/download"
GRCOV_VERSION: "0.8.19"
jobs:
my-job:
runs-on: ubuntu-22.04
steps:
- name: Checkout
uses: actions/checkout@v4
- name: install rust
uses: dtolnay/rust-toolchain@master
with:
components: rustfmt, clippy
toolchain: stable
- name: Install grcov
shell: bash
run: |
curl -L "$GRCOV_LINK/v$GRCOV_VERSION/grcov-x86_64-unknown-linux-musl.tar.bz2" |
tar xj -C $HOME/.cargo/bin
cargo xtask coverage install
- name: Build
shell: bash
run: cargo xtask build
- name: Tests
shell: bash
run: cargo xtask --enable-coverage test all
- name: Generate lcov.info
shell: bash
# /* is to exclude std library code coverage from analysis
run: cargo xtask coverage generate --ignore "/*,xtask/*,examples/*"
- name: Codecov upload lcov.info
uses: codecov/codecov-action@v4
with:
files: lcov.info
token: ${{ secrets.CODECOV_TOKEN }}
```
## Special command 'validate'
By convention this command is responsible to run all the checks, builds, and/or tests that validate the code
before opening a pull request or merge request.
The command `Validate` can been added via the macro `tracel_xtask_macros::commands` like the other commands.
By default all the checks from the `check` command are run as well as both unit and integration tests from
the `test` command.
You can make your own `handle_command` function if you need to perform more validations. Ideally this function
should only call the other commands `handle_command` functions.
For quick reference here is a simple example to perform all checks and test against the workspace:
```rust
pub fn handle_command(args: ValidateCmdArgs) -> anyhow::Result<()> {
let target = Target::Workspace;
let exclude = vec![];
let only = vec![];
// checks
[
CheckSubCommand::Audit,
CheckSubCommand::Format,
CheckSubCommand::Lint,
CheckSubCommand::Typos,
]
.iter()
.try_for_each(|c| {
super::check::handle_command(CheckCmdArgs {
target: target.clone(),
exclude: exclude.clone(),
only: only.clone(),
command: Some(c.clone()),
ignore_audit: args.ignore_audit,
})
})?;
// tests
super::test::handle_command(TestCmdArgs {
target: target.clone(),
exclude: exclude.clone(),
only: only.clone(),
threads: None,
jobs: None,
command: Some(TestSubCommand::All),
})?;
Ok(())
}
```
## Base commands list
### Check and Fix
`check` and `fix` contains the same subcommands to audit, format, lint or proofread a code base.
While the `check` command only reports issues the `fix` command attempt to fix them as they are encountered.
The `check` and `fix` commands are designed to help you maintain code quality during development.
They run various checks and fix issues, ensuring that your code is clean and follows best practices.
Each test can be executed separately or all of them can be executed sequentially using `all`.
Usage to lint the code base:
```sh
cargo xtask check lint
cargo xtask fix lint
cargo xtask fix all
```
### Running Tests
Testing is a crucial part of development, and the `test` command is designed to make this process easy.
This command makes the distinction between unit tests and integrations tests. [Unit tests][4] are inline tests under the
`src` directory of a crate. [Integration tests][3] are tests defined in files under the `tests` directory of a crate besides
the `src` directory.
Usage:
```sh
# execute workspace unit tests
cargo xtask test unit
# execute workspace integration tests
cargo xtask test integration
# execute workspace both unit tests and integration tests
cargo xtask test all
```
Note that documentation tests support is under the `doc` command.
### Documentation
Command to build and test the documentation in a workspace.
### Bumping Versions
This is a command reserved for repository maintainers.
The `bump` command is used to update the version numbers of all crates in the repository.
This is particularly useful when you're preparing for a new release and need to ensure that all crates have the correct version.
You can bump the version by major, minor, or patch levels, depending on the changes made.
For example, if you’ve made breaking changes, you should bump the major version.
For new features that are backwards compatible, bump the minor version.
For bug fixes, bump the patch version.
Usage:
```sh
cargo xtask bump <COMMAND>
```
### Publishing Crates
This is a command reserved for repository maintainers and is mainly in `publish` GitHub workflows.
This command automates the process of publishing crates to `crates.io`, the Rust package registry.
By specifying the name of the crate, `xtask` handles the publication process, ensuring that the crate is available for others to use.
Usage:
```sh
cargo xtask publish <NAME>
```
It is typical to use this command in a GitHub workflow that calls Tracel's reusable [publish-crate][8] workflow.
Here is a simple example with a workflow that publishes two crates A and B with A depending on B.
```yaml
name: publish all crates
on:
push:
tags:
- "v*"
jobs:
publish-B:
uses: tracel-ai/github-actions/.github/workflows/publish-crate.yml@v1
with:
crate: B
secrets:
CRATES_IO_API_TOKEN: ${{ secrets.CRATES_IO_API_TOKEN }}
# --------------------------------------------------------------------------------
publish-A:
uses: tracel-ai/github-actions/.github/workflows/publish-crate.yml@v1
with:
crate: A
needs:
- publish-B
secrets:
CRATES_IO_API_TOKEN: ${{ secrets.CRATES_IO_API_TOKEN }}
```
### Coverage
This command provide a subcommands to install the necessary dependencies for performing code coverage and a subcommand to generate the
coverage info file that can then be uploaded to codecov for instance. See dedicated section `Enable and generate coverage information`.
### Dependencies
Various additional commands about dependencies.
`deny` make sure that all dependencies meet requirements using [cargo-deny][5].
`unused` detects dependencies in the workspace that are not in ussed.
### Vulnerabilities
This command make it easier to execute sanitizers as described in [the Rust unstable book][6].
These sanitizers require a nightly toolchain.
```
Run the specified vulnerability check locally. These commands must be called with 'cargo +nightly'
Usage: xtask vulnerabilities <COMMAND>
Commands:
all Run all most useful vulnerability checks
address-sanitizer Run Address sanitizer (memory error detector)
control-flow-integrity Run LLVM Control Flow Integrity (CFI) (provides forward-edge control flow protection)
hw-address-sanitizer Run newer variant of Address sanitizer (memory error detector similar to AddressSanitizer, but based on partial hardware assistance)
kernel-control-flow-integrity Run Kernel LLVM Control Flow Integrity (KCFI) (provides forward-edge control flow protection for operating systems kerneljs)
leak-sanitizer Run Leak sanitizer (run-time memory leak detector)
memory-sanitizer Run memory sanitizer (detector of uninitialized reads)
mem-tag-sanitizer Run another address sanitizer (like AddressSanitizer and HardwareAddressSanitizer but with lower overhead suitable for use as hardening for production binaries)
nightly-checks Run nightly-only checks through cargo-careful `<https://crates.io/crates/cargo-careful>`
safe-stack Run SafeStack check (provides backward-edge control flow protection by separating stack into safe and unsafe regions)
shadow-call-stack Run ShadowCall check (provides backward-edge control flow protection - aarch64 only)
thread-sanitizer Run Thread sanitizer (data race detector)
help Print this message or the help of the given subcommand(s)
```
[1]: https://github.com/matklad/cargo-xtask
[2]: https://github.com/clap-rs/clap
[3]: https://doc.rust-lang.org/book/ch14-03-cargo-workspaces.html
[4]: https://doc.rust-lang.org/book/ch11-03-test-organization.html#integration-tests
[5]: https://doc.rust-lang.org/book/ch11-03-test-organization.html#unit-tests
[6]: https://embarkstudios.github.io/cargo-deny/
[7]: https://doc.rust-lang.org/beta/unstable-book/compiler-flags/sanitizer.html
[8]: https://github.com/tracel-ai/github-actions/blob/main/.github/workflows/publish-crate.yml