# Rust Mutation Testing
**Production-ready AST-based mutation testing for Rust - dogfooding PMAT itself!**
## Features
- 🎯 **80%+ mutation scores achievable** - Quantify test suite quality
- ⚡ **Fast generation** - Expected ~3ms for 50+ mutants
- 🔍 **Real test execution** - Works with cargo test
- 🧬 **8 mutation operators** - Binary, relational, logical, bitwise, range, pattern, method chain, borrow
- 🦀 **Rust-specific features** - Range operators (.., ..=), pattern matching, method chaining, borrows
- 📊 **Identifies test gaps** - Surviving mutants reveal actual weaknesses
- 🔄 **Full automation** - Source → mutants → tests → score
- 🏠 **Internal dogfooding** - PMAT tests itself using this system!
---
## Quick Start
### Prerequisites
```bash
# Install Rust toolchain
# Verify installation
cargo --version
rustc --version
```
### Example Workflow
```bash
cargo run --example rust_mutation_workflow --features rust-ast
```
**Output:**
```
🦀 Rust Mutation Testing Workflow
📝 Reading source file: calculator.rs
Size: 3142 bytes
🔧 Generating mutants...
Generated: 52 mutants
Time: 2.8ms
✅ Running baseline tests...
Baseline tests passed ✅
🧪 Testing 52 mutants...
[Progress...]
📊 Mutation Testing Results
Total Mutants: 52
Killed: 43 (82%)
Survived: 9 (17%)
Timeout/Error: 0
🎯 Mutation Score: 82% ✅ EXCELLENT!
```
---
## Mutation Operators
### 1. Binary Operator Replacement (AOR)
**Replaces:** `+, -, *, /, %`
```rust
// Original
pub fn add(a: i32, b: i32) -> i32 {
a + b
}
// Mutants
a - b // + → -
a * b // + → *
a / b // + → /
a % b // + → %
```
**Test to kill:**
```rust
#[test]
fn test_add() {
assert_eq!(add(2, 3), 5); // Kills all 4 mutants
assert_eq!(add(0, 5), 5); // Confirms + not *
assert_eq!(add(10, -3), 7); // Confirms + not -
}
```
### 2. Relational Operator Replacement (ROR)
**Replaces:** `<, >, <=, >=, ==, !=`
```rust
// Original
pub fn greater_than(a: i32, b: i32) -> bool {
a > b
}
// Mutants
a < b // > → <
a >= b // > → >=
a <= b // > → <=
a == b // > → ==
a != b // > → !=
```
**Test to kill:**
```rust
#[test]
fn test_greater_than() {
assert_eq!(greater_than(5, 3), true); // Kills most
assert_eq!(greater_than(3, 5), false); // Confirms direction
assert_eq!(greater_than(5, 5), false); // Kills >= mutant
}
```
### 3. Logical Operator Replacement (LOR)
**Replaces:** `&&, ||`
```rust
// Original
pub fn logical_and(a: bool, b: bool) -> bool {
a && b
}
// Mutants
**Test to kill:**
```rust
#[test]
fn test_logical_and() {
assert_eq!(logical_and(true, true), true); // Both pass
assert_eq!(logical_and(true, false), false); // Kills || mutant
assert_eq!(logical_and(false, false), false); // Edge case
}
```
### 4. Bitwise Operator Replacement (BOR)
**Replaces:** `&, |, ^, <<, >>`
```rust
// Original
pub fn bitwise_and(a: i32, b: i32) -> i32 {
a & b
}
// Mutants
```
**Test to kill:**
```rust
#[test]
fn test_bitwise_and() {
assert_eq!(bitwise_and(0b1100, 0b1010), 0b1000); // Kills all
assert_eq!(bitwise_and(0xFF, 0x0F), 0x0F); // Edge case
}
```
### 5. Range Operator Replacement (RANGEOR) - Rust-Specific 🦀
**Replaces:** `..` ↔ `..=`
```rust
// Original
pub fn exclusive_range_sum(start: i32, end: i32) -> i32 {
(start..end).sum()
}
// Mutant
(start..=end).sum() // .. → ..=
```
**Test to kill:**
```rust
#[test]
fn test_exclusive_range_sum() {
assert_eq!(exclusive_range_sum(0, 5), 10); // 0+1+2+3+4 = 10
assert_eq!(exclusive_range_sum(1, 4), 6); // 1+2+3 = 6
// If mutated to ..=, would get 15 and 10 instead
}
```
**Why it matters:** Off-by-one errors are the most common bugs in programming. This operator specifically targets them in Rust's elegant range syntax.
### 6. Pattern Match Detection (PMR) - Rust-Specific 🦀
**Detects:** `Some/None`, `Ok/Err` patterns
```rust
// Detected (not mutated - requires type inference)
pub fn unwrap_option(value: Option<i32>) -> i32 {
match value {
Some(x) => x,
None => 0,
}
}
```
**Why detection-only:** Swapping `Some` with `None` would require deep type inference to ensure semantic validity. Current implementation detects pattern matching for ML-based prioritization.
### 7. Method Chain Detection (MCR) - Rust-Specific 🦀
**Detects:** `.map`, `.filter`, `.collect`, `.fold`, etc.
```rust
// Detected (not mutated - requires type inference)
pub fn process_values(values: Vec<i32>) -> Vec<i32> {
values.iter().map(|x| x * 2).collect()
}
```
**Why detection-only:** Method chains involve complex type constraints. Swapping `.map` with `.filter` would require full type inference. Current implementation detects for analysis.
### 8. Borrow/Reference Detection (LBM) - Rust-Specific 🦀
**Detects:** `&`, `&mut`, `*` (dereference)
```rust
// Detected (not mutated - violates borrow checker)
pub fn borrow_immutable(value: &i32) -> i32 {
*value
}
pub fn borrow_mutable(value: &mut i32) {
*value += 1;
}
```
**Why detection-only:** Mutating borrow references would violate Rust's borrow checker rules. This is actually a **feature** - Rust's safety guarantees prevent entire classes of mutations that would be dangerous in other languages!
---
## Integration Guide
### 1. Basic API Usage
```rust
use pmat::services::mutation::{RustMutationGenerator, MutantStatus};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
// Read source
let source = std::fs::read_to_string("src/lib.rs")?;
// Generate mutants
let generator = RustMutationGenerator::with_default_operators();
let mut mutants = generator.generate_mutants(&source, "src/lib.rs")?;
println!("Generated {} mutants", mutants.len());
// Test each mutant (simplified)
for mutant in &mut mutants {
// Write mutant to temp file, run cargo test, check result
let tests_passed = test_mutant(&mutant.mutated_source).await?;
mutant.status = if tests_passed {
MutantStatus::Survived // Test gap!
} else {
MutantStatus::Killed // Good coverage
};
}
// Calculate score
let killed = mutants.iter().filter(|m| m.status == MutantStatus::Killed).count();
let total = mutants.len();
let score = (killed * 100) / total;
println!("Mutation Score: {}%", score);
Ok(())
}
```
### 2. Cargo Integration
Add to your `Cargo.toml`:
```toml
[dev-dependencies]
pmat = { version = "2.151", features = ["rust-ast"] }
tokio = { version = "1", features = ["full"] }
```
Create `tests/mutation.rs`:
```rust
#[cfg(test)]
mod mutation_tests {
use pmat::services::mutation::RustMutationGenerator;
#[test]
fn mutation_score_above_80_percent() {
let source = std::fs::read_to_string("src/lib.rs").unwrap();
let generator = RustMutationGenerator::with_default_operators();
let mutants = generator.generate_mutants(&source, "src/lib.rs").unwrap();
// Run mutations and calculate score
// assert!(score >= 80, "Mutation score too low: {}%", score);
}
}
```
### 3. CI/CD Integration
#### GitHub Actions
```yaml
name: Mutation Testing
on: [push, pull_request]
jobs:
mutation:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions-rs/toolchain@v1
with:
toolchain: stable
- name: Run Mutation Testing
run: |
cargo run --example rust_mutation_workflow --features rust-ast
- name: Check Mutation Score
run: |
# Parse score from output, fail if < 80%
SCORE=$(grep "Mutation Score:" mutation_output.txt | awk '{print $3}' | tr -d '%')
if [ "$SCORE" -lt 80 ]; then
echo "❌ Mutation score $SCORE% below threshold"
exit 1
fi
```
---
## Performance Benchmarks
| Small (500 LOC) | 25 | 1.2ms | 15s | 15.1s |
| Medium (2K LOC) | 50 | 2.8ms | 45s | 45.8s |
| Large (10K LOC) | 150 | 8.5ms | 3m 20s | 3m 20s |
**Note:** Test time dominates. Generation is negligible (<10ms even for large projects).
---
## Troubleshooting
### "No mutants generated"
**Cause:** Source file may not contain mutable operators.
**Fix:**
```bash
# Check if file has testable code
### "Baseline tests failed"
**Cause:** Your tests don't pass without mutations.
**Fix:**
```bash
cd fixtures/rust
cargo test
# Fix any failing tests first
```
### "All mutants survived"
**Cause:** Tests are too weak or don't exercise the code.
**Fix:** Add more comprehensive tests covering edge cases.
### "Compilation timeout"
**Cause:** Large codebase with many dependencies.
**Fix:** Run mutation testing on specific modules:
```rust
let source = std::fs::read_to_string("src/core.rs")?; // Just one module
```
---
## Best Practices
### 1. Start Small
Run mutation testing on critical modules first, not the entire codebase.
### 2. Incremental Adoption
```rust
// Week 1: Test core logic
mutation_test("src/core.rs");
// Week 2: Add business logic
mutation_test("src/business.rs");
// Week 3: Full coverage
mutation_test("src/");
```
### 3. Set Quality Gates
```rust
// In CI pipeline
if mutation_score < 80 {
fail_build();
}
```
### 4. Focus on Survivors
Don't chase 100%. Focus on **meaningful** surviving mutants that reveal actual test gaps.
### 5. Combine with Coverage
- **Line coverage** = "Did tests run this code?"
- **Mutation score** = "Did tests **verify** this code?"
Aim for: 90%+ line coverage + 80%+ mutation score
---
## Rust-Specific Considerations
### Memory Safety
Rust's borrow checker prevents entire classes of mutations that would be valid in C++:
```rust
// This mutation is IMPOSSIBLE in Rust (borrow checker prevents it)
fn invalid_mutation(value: &i32) -> &mut i32 {
value // ❌ Cannot convert & to &mut
}
```
This means Rust naturally has fewer dangerous mutations, which is a **feature**, not a limitation!
### Zero-Cost Abstractions
Rust's iterators compile to the same machine code as loops:
```rust
// These have identical performance
```
Mutation testing focuses on **logic**, not performance tradeoffs.
### Type Inference
Some mutations require full type inference (e.g., swapping `.map` with `.filter`). These are detected but not mutated in the current implementation.
---
## Comparison with Other Languages
| **Operators** | 8 (5 active) | 7 (5 active) | 9 (7 active) | 11 (8 active) | 7 (5 active) |
| **Language-Specific** | Ranges, patterns | Pointers, member access | List comp, decorators | Optional chaining, strict equality | Defer, channels |
| **Safety** | Borrow checker prevents invalid mutations | Manual memory management | Dynamic typing issues | Type narrowing | Goroutine safety |
| **Speed** | ~3ms/50 mutants | ~5ms/75 mutants | ~8ms/80 mutants | ~4ms/90 mutants | ~4ms/60 mutants |
---
## Examples
See `examples/rust_mutation_workflow.rs` for a complete, runnable example.
See `fixtures/rust/` for test fixtures:
- `calculator.rs` - 189 LOC with 25 functions covering all operators
- `tests/calculator_test.rs` - 326 LOC with 29 comprehensive tests
---
## Limitations
1. **Pattern matching** - Detection-only (requires type inference)
2. **Method chaining** - Detection-only (requires type inference)
3. **Borrow mutations** - Detection-only (would violate borrow checker)
4. **Macros** - Not yet supported
5. **Async code** - Experimental support
These limitations are by design - Rust's safety guarantees naturally prevent some classes of dangerous mutations!
---
## Roadmap
- [ ] Macro mutation support
- [ ] Async/await mutation operators
- [ ] Type-aware pattern matching mutations
- [ ] Integration with cargo-mutants
- [ ] Parallel test execution
- [ ] Incremental mutation (only changed code)
---
## FAQ
**Q: Why are some operators detection-only?**
A: Pattern matching and method chaining require full type inference to generate semantically valid mutations. Borrow mutations would violate Rust's safety guarantees.
**Q: How does this compare to cargo-mutants?**
A: PMAT uses tree-sitter AST parsing and focuses on Rust-specific features like ranges. cargo-mutants uses regex-based mutations and covers more general cases.
**Q: Can I use this in production?**
A: Yes! This is what we use to test PMAT itself (dogfooding). It's production-ready for Rust 2021 edition.
**Q: What mutation score should I aim for?**
A:
- **80%+** = Excellent test quality
- **60-80%** = Good, but room for improvement
- **<60%** = Weak test suite, add more tests
**Q: Is 100% possible?**
A: Yes, but not always worthwhile. Some survivors may be equivalent mutants or edge cases not worth testing.
---
## Contributing
Found a Rust-specific mutation operator we're missing? Submit a PR!
```bash
# 1. Add operator to rust_tree_sitter_mutations.rs
# 2. Update RustMutationGenerator
# 3. Add tests
# 4. Update this documentation
```
---
## License
MIT OR Apache-2.0
---
## Acknowledgments
- **tree-sitter-rust** - AST parsing
- **cargo test** - Test execution
- **Rust community** - Inspiration for Rust-specific operators
- **PMAT team** - Internal dogfooding and feedback
---
**Built with ❤️ and 🦀 by the PMAT team**
Mutation testing for Rust, by Rust developers, using Rust itself.