# Duroxide Stress Test Specification
## Overview
This document specifies the stress testing framework for Duroxide orchestration runtime. The stress tests validate both correctness and performance under various configurations and workloads.
## Test Dimensions
### 1. Test Scenarios
Stress tests are organized by scenario type, where each scenario represents a different orchestration pattern:
#### Current Scenarios
- **Parallel Orchestrations**: Fan-out/fan-in orchestration pattern with concurrent instance execution
- Each orchestration fans out to N activities and waits for all to complete
- Measures throughput, latency, and correctness
#### Planned Scenarios
- **Timer-Based Orchestrations**: Orchestrations with significant timer usage
- **Long-Running Orchestrations**: Continuously running orchestrations with periodic work
- **Sub-Orchestration Patterns**: Parent orchestrations spawning child orchestrations
- **Mixed Workloads**: Combination of different orchestration types running concurrently
### 2. Storage Providers
Tests run against multiple provider implementations:
- **In-Memory SQLite**: `SqliteProvider::new_in_memory()`
- Fastest execution, no I/O overhead
- Tests pure runtime performance
- **File-Based SQLite**: `SqliteProvider::new("sqlite:<path>", None)`
- Real-world persistence with WAL mode
- Tests I/O overhead and concurrency behavior
- **Custom Providers**: Extensible via `provider-test` feature
- Users can test their own provider implementations
- Enables benchmarking different storage backends
### 3. Concurrency Configurations
Each test runs with different runtime concurrency settings:
| 1/1 | 1 | 1 | Sequential processing, minimal contention |
| 2/2 | 2 | 2 | Balanced concurrency, default recommended |
| Higher configs | N/M | N/M | Test scalability limits |
## Test Output Format
### Console Output
Each test produces a summary table in the console:
```
=== Duroxide Stress Test Results ===
Test Scenario: Parallel Orchestrations
Duration: 10s
Max Concurrent Instances: 20
Tasks per Instance: 5
Activity Delay: 10ms
| In-Memory SQLite | 1/1 | 179 | 0 | 100.0% | 4.43 | 22.13 | 232ms |
| In-Memory SQLite | 2/2 | 257 | 0 | 100.0% | 6.66 | 33.28 | 150ms |
| File SQLite | 1/1 | 171 | 0 | 100.0% | 15.24 | 76.21 | 66ms |
| File SQLite | 2/2 | 267 | 0 | 100.0% | 7.75 | 38.77 | 129ms |
```
### Success Metrics
Each test validates:
1. **Success Rate**: `completed / (completed + failed) * 100`
- Target: 100% success rate
- Any failures indicate correctness issues
2. **Determinism**: Verify orchestrations produce identical results on repeated runs
- Test with same input data multiple times
- Assert output consistency
3. **Ordering**: Validate message processing order matches expectations
- For tests with ordering requirements
4. **Completion**: All launched orchestrations should complete within timeout
- No hanging or incomplete instances
5. **Memory**: Monitor for memory leaks over long runs
- Track memory usage trends
6. **Database Consistency**: For file-based providers, verify WAL integrity
- Ensure no corruption or lost writes
## Running Stress Tests
### Standard Provider Tests
Run all built-in SQLite provider tests:
```bash
cd sqlite-stress
cargo run --release --bin sqlite-stress
```
Or from the workspace root:
```bash
./run-stress-tests.sh [DURATION_SECS]
```
### Custom Provider Tests
To enable custom provider testing:
1. Add to `Cargo.toml`:
```toml
[dependencies]
duroxide = { path = "..", features = ["provider-test"] }
```
2. Implement the provider test adapter:
```rust
use duroxide::providers::Provider;
use duroxide_stress_tests::StressTestConfig;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let config = StressTestConfig {
max_concurrent: 20,
duration_secs: 10,
tasks_per_instance: 5,
activity_delay_ms: 10,
orch_concurrency: 2,
worker_concurrency: 2,
wait_timeout_secs: 60,
};
// Create your custom provider
let provider = Arc::new(MyCustomProvider::new().await?);
// Run stress test with custom provider
duroxide_stress_tests::run_with_provider(config, provider).await?;
Ok(())
}
```
3. Run your custom test:
```bash
cargo run --release --bin my_provider_stress_test
```
## Implementation Plan
### Phase 1: Refactor Current Tests
- [ ] Extract common test infrastructure
- [ ] Create `StressTestConfig` struct
- [ ] Implement `run_stress_test` generic function
- [ ] Add provider abstraction layer
### Phase 2: Add Provider Test Feature
- [ ] Add `provider-test` feature to `duroxide` crate
- [ ] Export provider testing utilities
- [ ] Create provider adapter trait
- [ ] Document custom provider integration
### Phase 3: Enhanced Reporting
- [ ] Add structured JSON output option
- [ ] Implement latency percentiles (P50, P95, P99)
- [ ] Add memory usage tracking
- [ ] Create comparison tables across configurations
### Phase 4: Additional Scenarios
- [ ] Timer-based orchestration stress test
- [ ] Long-running orchestration test
- [ ] Sub-orchestration stress test
- [ ] Mixed workload test
## Key Measurements
### Performance Metrics
- **Throughput**: Orchestrations completed per second
- **Activity Throughput**: Activities executed per second
- **Average Latency**: Mean time per orchestration completion
- **Tail Latency**: P95, P99 percentiles (planned)
- **Memory Usage**: Peak and average memory consumption (planned)
### Correctness Metrics
- **Success Rate**: Percentage of successful completions
- **Determinism**: Consistency across multiple runs
- **Data Integrity**: No corruption or lost data
- **Isolation**: Instance-level isolation maintained
## Success Criteria
### Minimum Requirements
- ✅ 100% success rate across all configurations
- ✅ Deterministic replay produces identical results
- ✅ No data corruption or loss
- ✅ Instance isolation maintained
- ✅ No memory leaks (pending implementation)
### Performance Targets
For parallel orchestrations with file-based SQLite:
- **Baseline (1/1)**: ≥ 10 orch/sec
- **Optimized (2/2)**: ≥ 15 orch/sec
- **Latency**: Average < 200ms per orchestration
## Future Enhancements
### Additional Dimensions
- **Load Testing**: Scale to 100+ concurrent instances
- **Duration Tests**: Run for extended periods (1+ hours)
- **Failure Injection**: Test behavior under failures
- **Network Latency**: Simulate network delays
- **Database Load**: Test with shared database under load
### Comparison Benchmarks
- Track performance trends over time
- Compare Duroxide performance to other frameworks
- Document regression detection criteria
### Continuous Integration
- Run stress tests as part of CI/CD pipeline
- Fail builds on performance regressions
- Store historical performance data
## Notes
- File-based SQLite uses WAL mode for better concurrency
- Database files are created in `/tmp` with unique names
- Each test run creates fresh database instances
- Tests run in release mode for accurate performance metrics
- Logging is set to INFO level to reduce noise
## Related Documentation
- **[Provider Testing Guide](provider-testing-guide.md)** - Complete instructions for testing custom providers
- **[Provider Implementation Guide](provider-implementation-guide.md)** - How to implement a custom provider