# Cleanroom Testing Framework
**Production-ready cleanroom testing framework using testcontainers following core team best practices.**
## Overview
The Cleanroom Testing Framework provides a comprehensive, production-ready testing environment using testcontainers with the following core team best practices:
- **Standardized testcontainers version (0.22)** across all projects
- **Singleton container pattern** for performance optimization
- **Container customizers** for flexible configuration
- **Proper lifecycle management** with RAII
- **Resource cleanup and error handling**
- **Performance monitoring and metrics collection**
- **Security boundaries and isolation**
- **Deterministic execution with fixed seeds**
## Validation Status
✅ **FULLY OPERATIONAL** - Comprehensive validation completed on 2025-10-13
| **Docker Integration** | ✅ Operational | 92% pass rate |
| **Testcontainers v0.25** | ✅ Active | Real containers |
| **Test Infrastructure** | ✅ Complete | 7+ test files |
| **ggen Integration** | ✅ Active | CLI, core, marketplace |
| **Production Readiness** | ✅ Approved | See validation report |
**Documentation**: See [CLEANROOM_OPERATIONAL_VALIDATION_REPORT.md](../docs/CLEANROOM_OPERATIONAL_VALIDATION_REPORT.md) for complete validation details.
**Quick Validation**:
```bash
# Validate Docker integration (92% pass rate)
bash ../scripts/validate-docker-integration.sh
# Quick Docker health check
bash ../scripts/quick-docker-check.sh
# Run cleanroom tests
cargo test
cargo test --test simple_testcontainer_test
cargo test --test testcontainer_e2e_test
```
## Features
### 🚀 Core Features
- **Singleton Containers**: Start containers once per test suite for performance
- **Resource Monitoring**: Track CPU, memory, disk, and network usage
- **Security Isolation**: Network, filesystem, and process isolation
- **Deterministic Execution**: Fixed seeds for reproducible tests
- **Coverage Tracking**: Track test coverage and execution paths
- **Snapshot Testing**: Capture and compare test outputs
- **Tracing & Observability**: Detailed tracing and metrics collection
- **Error Handling**: Comprehensive error handling and recovery
- **Performance Monitoring**: Real-time performance monitoring and alerting
### 🛡️ Security Features
- **Network Isolation**: Isolated network environments
- **Filesystem Isolation**: Secure filesystem boundaries
- **Process Isolation**: Isolated process execution
- **Data Redaction**: Automatic sensitive data redaction
- **Policy Enforcement**: Configurable security policies
### 📊 Monitoring & Observability
- **Performance Metrics**: Real-time performance monitoring
- **Resource Limits**: Configurable resource constraints
- **Tracing**: Distributed tracing support
- **Coverage Analysis**: Test coverage measurement
- **Snapshot Validation**: Automated snapshot testing
- **Error Tracking**: Comprehensive error tracking and reporting
## Quick Start
### Basic Usage
```rust
use cleanroom::{CleanroomEnvironment, CleanroomConfig};
use std::time::Duration;
#[tokio::test]
async fn test_my_application() {
// Create cleanroom environment with best practices
let config = CleanroomConfig::default();
let environment = CleanroomEnvironment::new(config).await.unwrap();
// Execute test with proper lifecycle management
let result = environment.execute_test("my_test", || {
// Your test logic here
Ok("test_passed")
}).await.unwrap();
// Cleanup is automatic via RAII
}
```
### Advanced Usage
```rust
use cleanroom::{
CleanroomEnvironment, CleanroomConfig, PostgresContainer, RedisContainer,
SecurityPolicy, PerformanceMonitoringConfig, ResourceLimits,
ContainerCustomizer, VolumeMount, PortMapping,
ContainerResourceLimits, HealthCheckConfig,
};
use std::time::Duration;
use std::collections::HashMap;
#[tokio::test]
async fn test_advanced_cleanroom() {
// Create advanced configuration
let mut config = CleanroomConfig::default();
// Configure security policy
config.security_policy.enable_network_isolation = true;
config.security_policy.enable_filesystem_isolation = true;
config.security_policy.enable_process_isolation = true;
config.security_policy.allowed_ports = vec![5432, 6379, 8080];
config.security_policy.enable_data_redaction = true;
config.security_policy.redaction_patterns = vec![
r"password\s*=\s*[^\s]+".to_string(),
r"token\s*=\s*[^\s]+".to_string(),
];
// Configure performance monitoring
config.performance_monitoring.enable_monitoring = true;
config.performance_monitoring.metrics_interval = Duration::from_secs(5);
config.performance_monitoring.thresholds.max_cpu_usage_percent = 80.0;
config.performance_monitoring.thresholds.max_memory_usage_bytes = 1024 * 1024 * 1024;
// Configure resource limits
config.resource_limits.max_cpu_usage_percent = 80.0;
config.resource_limits.max_memory_usage_bytes = 1024 * 1024 * 1024;
config.resource_limits.max_disk_usage_bytes = 10 * 1024 * 1024 * 1024;
config.resource_limits.max_container_count = 10;
// Configure deterministic execution
config.enable_deterministic_execution = true;
config.deterministic_seed = Some(42);
// Configure container customizers
let postgres_customizer = ContainerCustomizer {
name: "postgres_customizer".to_string(),
env_vars: HashMap::from([
("POSTGRES_DB".to_string(), "testdb".to_string()),
("POSTGRES_USER".to_string(), "testuser".to_string()),
("POSTGRES_PASSWORD".to_string(), "testpass".to_string()),
]),
volume_mounts: vec![
VolumeMount {
host_path: "/tmp/test".to_string(),
container_path: "/data".to_string(),
read_only: false,
},
],
port_mappings: vec![
PortMapping {
container_port: 5432,
host_port: Some(5433),
protocol: "tcp".to_string(),
},
],
resource_limits: ContainerResourceLimits {
cpu_limit: 0.5,
memory_limit_bytes: 512 * 1024 * 1024,
disk_limit_bytes: 1024 * 1024 * 1024,
network_bandwidth_limit: 100 * 1024 * 1024,
},
health_check: HealthCheckConfig {
command: "pg_isready -U testuser -d testdb".to_string(),
interval: Duration::from_secs(10),
timeout: Duration::from_secs(5),
retries: 3,
start_period: Duration::from_secs(30),
},
init_commands: vec![
"psql -U testuser -d testdb -c 'CREATE TABLE IF NOT EXISTS test_table (id SERIAL PRIMARY KEY);'".to_string(),
],
};
config.container_customizers.insert("postgres".to_string(), postgres_customizer);
// Create cleanroom environment
let environment = CleanroomEnvironment::new(config).await.unwrap();
// Get or create PostgreSQL container (singleton pattern)
let postgres = environment.get_or_create_container("postgres", || {
PostgresContainer::new(&environment.docker_client, "testdb", "testuser", "testpass")
}).await.unwrap();
// Wait for container to be ready
postgres.wait_for_ready().await.unwrap();
// Execute test with proper lifecycle management
let result = environment.execute_test("database_test", || {
// Your test logic here
Ok("test_passed")
}).await.unwrap();
// Cleanup is automatic via RAII
}
```
## Architecture
### Core Components
```mermaid
graph TB
A[CleanroomEnvironment] --> B[ContainerRegistry]
A --> C[PolicyEnforcement]
A --> D[DeterministicManager]
A --> E[CoverageTracker]
A --> F[SnapshotManager]
A --> G[TracingManager]
A --> H[ResourceLimits]
A --> I[RedactionManager]
A --> J[TestReport]
B --> K[PostgresContainer]
B --> L[RedisContainer]
B --> M[GenericContainer]
K --> N[ContainerWrapper]
L --> N
M --> N
```
### Container Management
The framework uses a singleton pattern for container management to optimize performance:
```rust
// Containers are created once and reused across tests
let postgres = environment.get_or_create_container("postgres", || {
PostgresContainer::new(&environment.docker_client, "testdb", "testuser", "testpass")
}).await.unwrap();
```
### Resource Monitoring
Real-time resource monitoring with configurable limits:
```rust
// Check resource limits
let result = environment.check_resource_limits().await;
assert!(result.is_ok());
// Get current metrics
let metrics = environment.get_metrics().await;
println!("CPU Usage: {:.1}%", metrics.resource_usage.peak_cpu_usage_percent);
println!("Memory Usage: {} bytes", metrics.resource_usage.peak_memory_usage_bytes);
```
### Security Features
Comprehensive security features with configurable policies:
```rust
// Configure security policy
config.security_policy.enable_network_isolation = true;
config.security_policy.enable_filesystem_isolation = true;
config.security_policy.enable_process_isolation = true;
config.security_policy.allowed_ports = vec![5432, 6379, 8080];
config.security_policy.enable_data_redaction = true;
config.security_policy.redaction_patterns = vec![
r"password\s*=\s*[^\s]+".to_string(),
r"token\s*=\s*[^\s]+".to_string(),
];
```
### Deterministic Execution
Fixed seed for reproducible tests:
```rust
// Configure deterministic execution
config.enable_deterministic_execution = true;
config.deterministic_seed = Some(42);
// Use deterministic random numbers
let random_value = environment.deterministic_manager.random().await;
```
### Coverage Tracking
Track test coverage and execution paths:
```rust
// Start coverage tracking
environment.coverage_tracker.start_test("test_name".to_string(), "test.rs".to_string()).await.unwrap();
// Record line coverage
environment.coverage_tracker.record_line_coverage("test_name", 10, true).await.unwrap();
// End coverage tracking
environment.coverage_tracker.end_test("test_name", TestStatus::Passed, 100).await.unwrap();
```
### Snapshot Testing
Capture and compare test outputs:
```rust
// Capture snapshot
environment.snapshot_manager.capture_snapshot(
"test_snapshot".to_string(),
"test content".to_string(),
SnapshotType::Text,
HashMap::new(),
).await.unwrap();
// Validate snapshot
let result = environment.snapshot_manager.validate_snapshot("test_snapshot", "test content").await;
assert!(result.is_ok());
```
### Tracing & Observability
Detailed tracing and metrics collection:
```rust
// Start span
let span_id = environment.tracing_manager.start_span("test_span".to_string(), None).await.unwrap();
// Add event
environment.tracing_manager.add_span_event(
"test_span",
"test_event".to_string(),
HashMap::from([("key".to_string(), "value".to_string())]),
).await.unwrap();
// End span
environment.tracing_manager.end_span("test_span", SpanStatus::Completed).await.unwrap();
```
### Data Redaction
Automatic sensitive data redaction:
```rust
// Test redaction
let content = "password=secret123 token=abc456";
let redacted = environment.redaction_manager.redact(content).await.unwrap();
assert!(redacted.contains("[REDACTED]"));
assert!(!redacted.contains("secret123"));
assert!(!redacted.contains("abc456"));
```
## Configuration
### CleanroomConfig
The main configuration structure:
```rust
pub struct CleanroomConfig {
pub enable_singleton_containers: bool,
pub container_startup_timeout: Duration,
pub test_execution_timeout: Duration,
pub enable_deterministic_execution: bool,
pub deterministic_seed: Option<u64>,
pub enable_coverage_tracking: bool,
pub enable_snapshot_testing: bool,
pub enable_tracing: bool,
pub resource_limits: ResourceLimits,
pub security_policy: SecurityPolicy,
pub performance_monitoring: PerformanceMonitoringConfig,
pub container_customizers: HashMap<String, ContainerCustomizer>,
}
```
### SecurityPolicy
Security configuration:
```rust
pub struct SecurityPolicy {
pub enable_network_isolation: bool,
pub enable_filesystem_isolation: bool,
pub enable_process_isolation: bool,
pub allowed_ports: Vec<u16>,
pub blocked_addresses: Vec<String>,
pub enable_data_redaction: bool,
pub redaction_patterns: Vec<String>,
pub enable_audit_logging: bool,
pub security_level: SecurityLevel,
}
```
### PerformanceMonitoringConfig
Performance monitoring configuration:
```rust
pub struct PerformanceMonitoringConfig {
pub enable_monitoring: bool,
pub metrics_interval: Duration,
pub thresholds: PerformanceThresholds,
pub enable_profiling: bool,
pub enable_memory_tracking: bool,
}
```
### ResourceLimits
Resource limits configuration:
```rust
pub struct ResourceLimits {
pub max_cpu_usage_percent: f64,
pub max_memory_usage_bytes: u64,
pub max_disk_usage_bytes: u64,
pub max_network_bandwidth_bytes_per_sec: u64,
pub max_container_count: u32,
pub max_test_execution_time: Duration,
pub enable_resource_monitoring: bool,
pub resource_cleanup_timeout: Duration,
}
```
## Container Types
### PostgreSQL Container
```rust
let postgres = PostgresContainer::new(&docker_client, "testdb", "testuser", "testpass").unwrap();
postgres.wait_for_ready().await.unwrap();
// Execute SQL
let result = postgres.execute_sql("SELECT 1;").await.unwrap();
// Get database size
let size = postgres.get_database_size().await.unwrap();
// Get active connections
let connections = postgres.get_active_connections().await.unwrap();
```
### Redis Container
```rust
let redis = RedisContainer::new(&docker_client, None).unwrap();
redis.wait_for_ready().await.unwrap();
// Set key-value pair
redis.set("key", "value").await.unwrap();
// Get value
let value = redis.get("key").await.unwrap();
// Delete key
redis.del("key").await.unwrap();
```
### Generic Container
```rust
let container = GenericContainer::new(&docker_client, "test", "alpine", "latest").unwrap();
container.wait_for_ready().await.unwrap();
// Execute command
let result = container.execute_command(vec!["echo".to_string(), "hello".to_string()]).await.unwrap();
```
## Error Handling
The framework provides comprehensive error handling:
```rust
use cleanroom::CleanroomError;
// Handle different error types
match result {
Ok(value) => println!("Success: {:?}", value),
Err(CleanroomError::ContainerError(msg)) => println!("Container error: {}", msg),
Err(CleanroomError::NetworkError(msg)) => println!("Network error: {}", msg),
Err(CleanroomError::ResourceLimitExceeded(msg)) => println!("Resource limit exceeded: {}", msg),
Err(CleanroomError::Timeout(msg)) => println!("Timeout: {}", msg),
Err(e) => println!("Other error: {}", e),
}
```
## Testing
### Running Tests
```bash
# Run all tests
cargo test
# Run integration tests
cargo test --test integration_tests
# Run container tests
cargo test --test container_tests
# Run with Docker (requires Docker daemon)
cargo test --test integration_tests -- --ignored
cargo test --test container_tests -- --ignored
```
### Test Structure
```
cleanroom/
├── src/
│ ├── cleanroom.rs # Core cleanroom environment
│ ├── containers.rs # Container implementations
│ ├── error.rs # Error types
│ ├── policy.rs # Policy enforcement
│ ├── determinism.rs # Deterministic execution
│ ├── coverage.rs # Coverage tracking
│ ├── snapshots.rs # Snapshot testing
│ ├── tracing.rs # Tracing and observability
│ ├── limits.rs # Resource limits
│ ├── redaction.rs # Data redaction
│ └── report.rs # Test reporting
├── tests/
│ ├── integration_tests.rs # Integration tests
│ └── container_tests.rs # Container tests
└── README.md # This file
```
## Best Practices
### 1. Use Singleton Containers
```rust
// Good: Use singleton pattern
let postgres = environment.get_or_create_container("postgres", || {
PostgresContainer::new(&environment.docker_client, "testdb", "testuser", "testpass")
}).await.unwrap();
// Avoid: Creating new containers for each test
let postgres = PostgresContainer::new(&docker_client, "testdb", "testuser", "testpass").unwrap();
```
### 2. Configure Resource Limits
```rust
// Good: Set appropriate resource limits
config.resource_limits.max_cpu_usage_percent = 80.0;
config.resource_limits.max_memory_usage_bytes = 1024 * 1024 * 1024;
config.resource_limits.max_disk_usage_bytes = 10 * 1024 * 1024 * 1024;
// Avoid: Using default limits without consideration
```
### 3. Enable Security Features
```rust
// Good: Enable security features
config.security_policy.enable_network_isolation = true;
config.security_policy.enable_filesystem_isolation = true;
config.security_policy.enable_process_isolation = true;
config.security_policy.enable_data_redaction = true;
// Avoid: Disabling security features without good reason
```
### 4. Use Deterministic Execution
```rust
// Good: Use deterministic execution for reproducible tests
config.enable_deterministic_execution = true;
config.deterministic_seed = Some(42);
// Avoid: Non-deterministic execution unless necessary
```
### 5. Monitor Performance
```rust
// Good: Enable performance monitoring
config.performance_monitoring.enable_monitoring = true;
config.performance_monitoring.metrics_interval = Duration::from_secs(5);
// Check resource limits
let result = environment.check_resource_limits().await;
assert!(result.is_ok());
```
### 6. Handle Errors Properly
```rust
// Good: Handle errors with specific error types
match result {
Ok(value) => println!("Success: {:?}", value),
Err(CleanroomError::ContainerError(msg)) => {
eprintln!("Container error: {}", msg);
// Handle container-specific error
},
Err(CleanroomError::ResourceLimitExceeded(msg)) => {
eprintln!("Resource limit exceeded: {}", msg);
// Handle resource limit error
},
Err(e) => {
eprintln!("Unexpected error: {}", e);
// Handle unexpected error
},
}
// Avoid: Ignoring errors or using generic error handling
```
### 7. Use RAII for Cleanup
```rust
// Good: Use RAII guard for automatic cleanup
let _guard = CleanroomGuard::new(environment_arc.clone());
// Avoid: Manual cleanup without RAII
```
## Performance Considerations
### Container Startup Time
- Use singleton containers to reduce startup time
- Configure appropriate startup timeouts
- Use health checks to ensure containers are ready
### Memory Usage
- Set appropriate memory limits
- Monitor memory usage during tests
- Use memory-efficient container images
### CPU Usage
- Set appropriate CPU limits
- Monitor CPU usage during tests
- Use CPU-efficient container images
### Disk Usage
- Set appropriate disk limits
- Monitor disk usage during tests
- Use disk-efficient container images
### Network Usage
- Set appropriate network limits
- Monitor network usage during tests
- Use network-efficient container images
## Troubleshooting
### Common Issues
#### Docker Not Running
```bash
# Error: Docker daemon is not running
# Solution: Start Docker daemon
sudo systemctl start docker
```
#### Port Conflicts
```bash
# Error: Port already in use
# Solution: Check for conflicting services
```
#### Container Startup Failures
```bash
# Error: Container failed to start
# Solution: Check Docker logs
docker logs <container_id>
```
#### Test Timeouts
```bash
# Error: Test timeout
# Solution: Increase timeout or check container health
export TEST_TIMEOUT=600
```
### Debug Mode
#### Enable Verbose Logging
```bash
RUST_LOG=debug cargo test --test integration_tests
```
#### Container Inspection
```bash
# List running containers
docker ps
# Inspect container logs
docker logs <container_id>
# Execute commands in container
docker exec -it <container_id> /bin/bash
```
#### Network Debugging
```bash
# Check container networking
docker network ls
docker network inspect <network_id>
# Test connectivity
docker exec <container_id> ping <host>
```
## Contributing
### Development Setup
1. Clone the repository
2. Install dependencies: `cargo build`
3. Run tests: `cargo test`
4. Run integration tests: `cargo test --test integration_tests -- --ignored`
### Code Style
- Follow Rust conventions
- Use `cargo fmt` for formatting
- Use `cargo clippy` for linting
- Write comprehensive tests
- Document public APIs
### Pull Request Process
1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Ensure all tests pass
6. Submit a pull request
## License
This project is licensed under the MIT License - see the LICENSE file for details.
## Acknowledgments
- [testcontainers-rs](https://github.com/testcontainers/testcontainers-rs) for container testing inspiration
- [Docker](https://www.docker.com/) for containerization platform
- [Rust](https://www.rust-lang.org/) for the programming language
- [Tokio](https://tokio.rs/) for async runtime