allsource-core 0.19.1

# AllSource Core - Feature Showcase

## Overview

AllSource Core is a **production-ready, enterprise-grade event sourcing platform** built in Rust with comprehensive security, performance, and scalability features.

---

## 🎯 Core Features

### Event Store Architecture
- **High-Performance Event Sourcing**: Optimized append-only event store
- **Multi-Tenancy**: Complete tenant isolation with quotas and limits
- **Schema Registry**: Versioned event schemas with compatibility modes
- **Projections**: Real-time and batch materialized views
- **Event Replay**: Point-in-time recovery and debugging
- **Snapshots**: Optimized aggregate reconstruction
- **Write-Ahead Log (WAL)**: Durable, crash-resistant writes
- **Compaction**: Automatic cleanup and optimization

### Data Storage & Processing
- **Apache Arrow**: Zero-copy columnar format for lightning-fast queries
- **Apache Parquet**: Efficient long-term storage with compression
- **Apache DataFusion**: SQL query engine for analytics
- **Multiple Storage Backends**:
  - In-memory (development/testing)
  - PostgreSQL (production)
  - RocksDB (high-performance, embedded)

### Performance
- **Lock-Free Data Structures**: Crossbeam queues for concurrency
- **Parallel Processing**: Multi-threaded event ingestion
- **Batch Operations**: Bulk event processing
- **Connection Pooling**: Optimized database connections
- **Compression**: LZ4 and GZIP support
- **Benchmarked**: Comprehensive performance testing suite

---

##  🔒 Advanced Security Features

### 1. ML-Based Anomaly Detection ✨

**Real-time threat detection using behavioral analysis and statistical modeling**

#### Detection Types
- **Brute Force Attacks**: 5+ failed logins in 15 minutes
- **Unusual Access Patterns**: Outside typical hours/actions for user
- **Privilege Escalation**: Unauthorized attempts to gain elevated access
- **Data Exfiltration**: 5x above normal query rates
- **Velocity Anomalies**: Impossibly fast actions (20+ in 10 seconds)

#### Features
- Behavioral baselining per user/tenant
- Statistical anomaly scoring (0.0-1.0 confidence)
- Automated recommendations (Monitor, Alert, RequireMFA, Block, RevokeAccess)
- Profile building with historical data
- Configurable sensitivity and thresholds

#### Usage
```rust
let detector = AnomalyDetector::new(config);
let result = detector.analyze_event(&audit_event)?;

if result.is_anomalous {
    match result.recommended_action {
        RecommendedAction::Block => block_user(),
        RecommendedAction::Alert => send_security_alert(),
        // ...
    }
}
```

---

### 2. Field-Level Encryption ✨

**Transparent AES-256-GCM encryption for sensitive data**

#### Features
- AES-256-GCM authenticated encryption
- Per-field encryption keys
- Automatic key rotation without downtime
- Multi-version key support (seamless migration)
- Envelope encryption pattern
- Base64 encoding for storage

#### Key Rotation
```rust
let encryption = FieldEncryption::new(config)?;

// Encrypt with current key
let encrypted = encryption.encrypt_string("sensitive", "field")?;

// Rotate keys (old data still decryptable)
encryption.rotate_keys()?;

// New encryptions use new key
let new_encrypted = encryption.encrypt_string("more data", "field")?;

// Both versions decrypt correctly
let old_data = encryption.decrypt_string(&encrypted)?;
let new_data = encryption.decrypt_string(&new_encrypted)?;
```

---

### 3. HSM/KMS Integration ✨

**Enterprise key management with external providers**

#### Supported Providers
- **AWS KMS**: Amazon's managed key service
- **Google Cloud KMS**: GCP key management
- **Azure Key Vault**: Microsoft's secure key storage
- **HashiCorp Vault**: Open-source secret management
- **PKCS#11 HSM**: Hardware security modules
- **Local KMS**: Testing and development

#### Envelope Encryption
```rust
let kms = KmsManager::new(config)?;

// Create master key
let master_key = kms.client().create_key(
    "master-key".to_string(),
    KeyPurpose::DataEncryption,
    KeyAlgorithm::Aes256Gcm,
).await?;

// Envelope encryption (DEK + Master Key)
let encrypted = kms.envelope_encrypt(&master_key.key_id, data).await?;
let decrypted = kms.envelope_decrypt(&encrypted).await?;

// Key rotation
kms.client().rotate_key(&master_key.key_id).await?;
```

---

### 4. Adaptive Rate Limiting ✨

**ML-based automatic rate limit adjustment**

#### Adaptation Strategies
- **Learning-Based**: Adjust based on historical usage (±30% default)
- **Anomaly-Based**: Throttle on 3x normal rate detection
- **Load-Based**: Reduce limits when CPU/memory > 80%
- **Pattern Prediction**: Proactive increases at known peak times
- **Safety Bounds**: Configurable min/max limits

#### Dynamic Adjustment
```rust
let limiter = AdaptiveRateLimiter::new(config);

// Check limit (automatically adjusted)
let result = limiter.check_adaptive_limit("tenant-id")?;

// Record system metrics
limiter.record_system_load(SystemLoad {
    cpu_usage: 0.85,
    memory_usage: 0.78,
    active_connections: 5000,
    queue_depth: 1200,
});

// Update limits based on patterns and load
limiter.update_adaptive_limits()?;
```

---

### 5. Security Automation & CI/CD Scanning ✨

**Automated security validation for continuous deployment**

#### Scan Types
- **Dependency Vulnerabilities**: cargo audit integration
- **Secret Detection**: Hardcoded credentials, API keys, passwords
- **SAST**: Static analysis with cargo clippy
- **License Compliance**: Restricted license detection

#### CI/CD Integration
```rust
let scanner = SecurityScanner::new(config);
let result = scanner.run_full_scan()?;

match result.status {
    ScanStatus::Pass => continue_deployment(),
    ScanStatus::Fail => {
        for finding in result.findings {
            if finding.severity == Severity::Critical {
                abort_deployment();
            }
        }
    }
}

// Generate workflows
let workflow = CiCdIntegration::generate_github_actions_workflow();
fs::write(".github/workflows/security.yml", workflow)?;
```

---

## 🛡️ Standard Security Features

### Authentication
- **JWT Tokens**: Stateless authentication with configurable expiry
- **API Keys**: Service-to-service authentication
- **Password Hashing**: Argon2 with automatic salt generation
- **Token Refresh**: Secure token rotation
- **Session Management**: Configurable timeout and limits

### Authorization
- **RBAC**: Role-based access control (Admin, Developer, Analyst, Viewer)
- **Permissions**: Granular per-resource permissions
- **Tenant Isolation**: Complete data separation
- **Resource-Level Access**: Fine-grained control

### Rate Limiting
- **Per-Tenant Limits**: Configurable request quotas
- **Token Bucket Algorithm**: Smooth rate limiting
- **Distributed Rate Limiting**: Cross-instance coordination
- **Burst Handling**: Allow temporary spikes

### IP Filtering
- **Global Allow/Block Lists**: System-wide IP control
- **Per-Tenant Lists**: Tenant-specific IP restrictions
- **CIDR Support**: Network range filtering
- **Priority Rules**: Block overrides allow

### Audit Logging
- **Comprehensive Events**: All security-relevant actions logged
- **Tamper-Proof**: Append-only, immutable audit trail
- **Tenant Isolation**: Separate logs per tenant
- **Structured Format**: JSON for easy analysis
- **Retention Policies**: Configurable storage duration

### Security Headers
- **HSTS**: Force HTTPS connections
- **CSP**: Content Security Policy
- **X-Frame-Options**: Clickjacking protection
- **X-Content-Type-Options**: MIME sniffing prevention
- **Request ID**: Tracking and correlation

---

## 📊 Analytics & Monitoring

### Metrics (Prometheus)
- Request rates and latency
- Event ingestion throughput
- Storage utilization
- Cache hit rates
- Error rates by type
- Per-tenant metrics

### Health Checks
- Liveness probes
- Readiness probes
- Dependency health
- Resource availability

### Observability
- Structured logging (JSON)
- Distributed tracing support
- Request correlation IDs
- Performance profiling

---

## 🚀 Operations

### Deployment
- **Docker**: Multi-stage builds, minimal images
- **Kubernetes**: Ready for k8s deployment
- **Horizontal Scaling**: Stateless design
- **High Availability**: No single point of failure

### Backup & Recovery
- **Point-in-Time Recovery**: Event replay from any timestamp
- **Snapshots**: Fast state restoration
- **Cross-Region Replication**: Disaster recovery
- **Automated Backups**: Scheduled backup jobs

### Configuration
- **Environment Variables**: 12-factor app compliant
- **Configuration Files**: TOML/JSON support
- **Runtime Reconfiguration**: Dynamic updates
- **Secrets Management**: Integration with secret stores

---

## 🧪 Quality Assurance

### Testing
- **Unit Tests**: Comprehensive test coverage
- **Integration Tests**: End-to-end scenarios
- **Security Tests**: All security features validated
- **Performance Benchmarks**: Criterion-based benchmarking
- **Stress Tests**: 7-day continuous operation test

### Code Quality
- **Clippy Lints**: Strict Rust best practices
- **Rustfmt**: Consistent code formatting
- **Documentation**: Inline docs for all public APIs
- **Examples**: Real-world usage examples

---

## 📈 Performance Characteristics

### Event Ingestion
- **Throughput**: 10,000+ events/second (single instance)
- **Latency**: < 10ms p99 (in-memory)
- **Batching**: Up to 1000 events per batch
- **Parallelism**: Multi-threaded processing

### Query Performance
- **Point Queries**: < 1ms (indexed)
- **Range Queries**: Optimized with Apache Arrow
- **Aggregations**: Parallel execution with DataFusion
- **Projections**: Real-time updates

### Resource Usage
- **Memory**: Configurable cache sizes
- **CPU**: Multi-core utilization
- **Disk**: Compressed storage with Parquet
- **Network**: Efficient serialization

---

## 🔧 Development Experience

### Developer Tools
- **CLI Admin Tool**: Interactive management
- **REST API**: Full feature access
- **WebSocket Subscriptions**: Real-time event streaming
- **SQL Analytics**: DataFusion query engine

### Libraries & SDKs
- **Rust Library**: Core event store library
- **gRPC API**: Language-agnostic interface
- **Arrow Flight**: High-performance data transfer

---

## 📦 Package Contents

```
allsource-core/
├── src/
│   ├── security/          # Advanced security features
│   │   ├── anomaly_detection.rs    # ML-based threat detection
│   │   ├── encryption.rs           # Field-level encryption
│   │   ├── kms.rs                  # HSM/KMS integration
│   │   ├── adaptive_rate_limit.rs  # Adaptive rate limiting
│   │   └── automation.rs           # Security scanning
│   ├── auth/              # Authentication & authorization
│   ├── domain/            # Domain model
│   ├── infrastructure/    # Storage & external services
│   ├── application/       # Use cases & services
│   └── api/               # REST & gRPC APIs
├── examples/              # Comprehensive examples
├── tests/                 # Integration & stress tests
├── benches/               # Performance benchmarks
└── docs/                  # Additional documentation
```

---

## 🎓 Getting Started

### Integration Example
```rust
use allsource_core::{
    EventStore,
    security::{AnomalyDetector, FieldEncryption, AdaptiveRateLimiter},
};

#[tokio::main]
async fn main() -> Result<()> {
    // Initialize store with all security features
    let store = EventStore::new(config);
    let detector = AnomalyDetector::new(anomaly_config);
    let encryption = FieldEncryption::new(enc_config)?;
    let rate_limiter = AdaptiveRateLimiter::new(rate_config);

    // Your application logic with enterprise-grade security
    Ok(())
}
```

---

## 📚 Documentation

- **[SECURITY.md](SECURITY.md)**: Comprehensive security guide
- **[Examples](examples/)**: Working code examples
- **[API Docs](https://docs.rs/allsource-core)**: Complete API reference
- **[Architecture](docs/architecture.md)**: System design

---

## 🏆 Production Ready

✅ **Battle-tested** security features
✅ **Comprehensive** test suite
✅ **Documented** APIs and examples
✅ **Performant** under load
✅ **Scalable** architecture
✅ **Observable** with metrics
✅ **Maintainable** code quality

---

**AllSource Core** - Enterprise-Grade Event Sourcing with Advanced Security

*Built with ❤️ in Rust*