# IronDrop Architecture Documentation v2.7.0
## Overview
IronDrop is a file server written in Rust. It uses only the standard library for networking and file I/O (no external HTTP framework). This document provides an overview of the system architecture, component interactions, and implementation details.
## System Architecture
```
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ CLI Parser │───▶│ Server Init │───▶│Custom Thread Pool│
│ (cli.rs) │ │ (main.rs) │ │ (server.rs) │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│
▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Template Engine │◀───│ HTTP Handler │◀───│ Request Router │
│ (templates.rs) │ │ (response.rs) │ │ (http.rs) │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │ │
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Static Assets │ │ File System │ │Upload & Multipart│
│ (templates/*) │ │ (fs.rs) │ │upload.rs+multipart│
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │
▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Downloads │ │ Uploads │ │ Search Engine │
│ Range Requests │ │ Direct Streaming │ │Ultra-compact │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│
▼
┌─────────────────┐
│Security & Monitor│
│ Rate Limit+Stats │
└─────────────────┘
```
## Core Modules
### 1. **Entry Point & Configuration**
- **`main.rs`** (6 lines): Simple entry point that calls `irondrop::run()`
- **`lib.rs`** (56 lines): Library initialization, logging setup, and server bootstrap
- **`cli.rs`** (200+ lines): Command-line interface with comprehensive validation
- **`config/mod.rs`**: Configuration system with hierarchical precedence (CLI > INI > defaults)
- **`config/ini_parser.rs`**: Zero-dependency INI parser for configuration files
### 2. **HTTP Processing Layer**
- **`server.rs`**: Custom thread pool implementation with rate limiting and connection management
- **`http.rs`**: HTTP request parsing, routing, connection handling, and static asset serving
- **`response.rs`**: HTTP response building, MIME type detection, and error page generation
- **`router.rs`**: Simple HTTP router with exact and prefix path matching
- **`handlers.rs`**: Internal route handlers for health checks, status, uploads, and monitoring
- **`middleware.rs`**: Authentication middleware with Basic Auth support
### 3. **File Operations**
- **`fs.rs`**: Directory listing generation, file details, and file system interactions
- **`upload.rs`**: Direct upload handler with memory/disk streaming and atomic operations
### 4. **Search System**
- **`search.rs`**: Search system; ultra-compact mode with hierarchical path storage and string pooling
### 5. **Template System**
- **`templates.rs`**: Native template engine with embedded assets and variable interpolation
- **`templates/directory/`**: Directory listing templates (HTML, CSS, JS)
- **`templates/upload/`**: File upload templates (HTML, CSS, JS)
- **`templates/error/`**: Error page templates (HTML, CSS, JS)
- **`templates/monitor/`**: Monitoring dashboard templates
### 6. **Support Systems**
- **`error.rs`**: Comprehensive error types including upload-specific errors
- **`utils.rs`**: Utility functions for path handling, URL parsing, and encoding
## Request Processing Flow
```
HTTP Request
│
▼
┌─────────────────┐
│ Rate Limiting │───[Fail]───▶ 429 Too Many Requests
│ Check │
└─────────────────┘
│ [Pass]
▼
┌─────────────────┐
│ Authentication │───[Fail]───▶ 401 Unauthorized
│ Middleware │
└─────────────────┘
│ [Pass]
▼
┌─────────────────┐
│ HTTP Router │
│ (router.rs) │
└─────────────────┘
│
┌───────────┬───────────┼───────────┬───────────┬───────────┐
│ │ │ │ │ │
▼ ▼ ▼ ▼ ▼ ▼
[Static Assets] [Health] [Upload API] [File Sys] [Search API] [Monitor]
/_irondrop/ /_irondrop/health /_irondrop/ Directory /_irondrop/ /monitor
/static/* /upload Listing /search
│ │ │ │ │ │
▼ ▼ ▼ ▼ ▼ ▼
Serve CSS/JS JSON Status Process Upload Path Check Search Engine Dashboard
│ │ │
[Pass] │ [Fail] │ ▼
▼ ▼ JSON Results
Resource Type 403 Forbidden
Detection
│
┌─────────────┼─────────────┐
│ │ │
▼ ▼ ▼
[Directory] [File] [Upload UI]
│ │ │
▼ ▼ ▼
Template-based Stream File Upload Interface
Listing Content Processing
```
## File Structure
```
src/
├── main.rs # Entry point (6 lines)
├── lib.rs # Library initialization (56 lines)
├── cli.rs # CLI interface with validation (200+ lines)
├── config/ # Configuration system
│ ├── mod.rs # Config struct and loading logic
│ └── ini_parser.rs # Zero-dependency INI parser
├── server.rs # Thread pool + rate limiting (400+ lines)
├── http.rs # HTTP parsing + connection handling (600+ lines)
├── router.rs # HTTP routing system (100+ lines)
├── handlers.rs # Internal route handlers (200+ lines)
├── middleware.rs # Authentication middleware (100+ lines)
├── templates.rs # Template engine with embedded assets (300+ lines)
├── fs.rs # File system operations (200+ lines)
├── response.rs # Response building + MIME detection (400+ lines)
├── upload.rs # Direct upload handler with streaming (500+ lines)
├── search.rs # Ultra-compact search engine (400+ lines)
├── error.rs # Comprehensive error types (100+ lines)
└── utils.rs # Utility functions for paths and encoding
templates/
├── directory/ # Directory listing UI
│ ├── index.html # HTML structure with search
│ ├── styles.css # Professional dark theme
│ └── script.js # Interactive browsing + search
├── upload/ # Upload interface
│ ├── page.html # Standalone upload page
│ ├── form.html # Reusable upload component
│ ├── styles.css # Upload UI styling
│ └── script.js # Direct binary upload logic
├── error/ # Error pages
│ ├── page.html # Error page template
│ └── styles.css # Error page styling
└── monitor/ # Monitoring dashboard
├── page.html # Dashboard template
├── styles.css # Dashboard styling
└── script.js # Real-time metrics updates
└── error/ # Error pages
├── page.html # Error page structure
├── styles.css # Error styling
└── script.js # Error page enhancements
tests/
├── integration_test.rs # Core server tests (19 tests)
├── integration_test.rs # Auth + security tests (6 tests)
├── edge_case_test.rs # Upload edge cases (10 tests)
├── memory_optimization_test.rs # Memory efficiency (6 tests)
├── performance_test.rs # Upload performance (5 tests)
├── stress_test.rs # Stress testing (4 tests)
├── multipart_test.rs # Multipart parser tests (7 tests)
├── ultra_compact_test.rs # Search engine tests (4 tests)
├── template_embedding_test.rs # Template system tests (3 tests)
├── test_upload.sh # End-to-end upload testing
├── test_1gb_upload.sh # Large file upload testing
└── test_executable_portability.sh # Portability validation
Total: 199 tests across 16 test files
```
## Search System Architecture
### Overview
IronDrop features a sophisticated dual-mode search system designed for both efficiency and scalability, with support for directories containing millions of files while maintaining low memory usage.
### Search Implementation Modes
#### 1. **Standard Search Engine (`search.rs`)**
- **Target**: Directories with up to 100K files
- **Memory Usage**: ~10MB for 10K files
- **Features**:
- LRU cache with 5-minute TTL
- Thread-safe operations with `Arc<Mutex<>>`
- Fuzzy search with relevance scoring
- Real-time indexing with background updates
- Full-text search with token matching
#### 2. **Ultra-Compact Search (`ultra_compact_search.rs`)**
- **Target**: Directories with 10M+ files
- **Memory Usage**: <100MB for 10M files (11 bytes per entry)
- **Features**:
- Hierarchical path storage with parent references
- Unified string pool with binary search
- Bit-packed metadata (size, timestamps, flags)
- Cache-aligned structures for CPU optimization
- Radix-accelerated indexing
### Memory Optimization Techniques
```
Standard Entry (24 bytes): Ultra-Compact Entry (11 bytes):
┌────────────────────┐ ┌─────────────────┐
│ Full Path (String) │ │ Name Offset (3) │
│ Name (String) │ │ Parent ID (3) │
│ Size (u64) │ │ Size Log2 (1) │
│ Modified (u64) │ │ Packed Data (4) │
│ Flags (u32) │ └─────────────────┘
└────────────────────┘ 58% memory reduction
```
### Search Performance Characteristics
| 1K files | <1ms | <1ms |
| 10K files | 2-5ms | 1-3ms |
| 100K files | 10-20ms | 5-10ms |
| 1M files | N/A | 20-50ms |
| 10M files | N/A | 100-200ms |
### Search API Integration
The search system integrates with the HTTP layer through dedicated endpoints:
- **`GET /_irondrop/search?q=query`**: Primary search interface
- **Frontend Integration**: Real-time search with 300ms debouncing
- **Result Pagination**: Configurable limits and offsets
- **JSON Response Format**: Structured results with metadata
### Caching Strategy
```
Request → Cache Check → Hit: Return Cached Results
│
└─ Miss → Index Search → Cache Store → Return Results
```
- **LRU Eviction**: Least recently used entries removed first
- **TTL Expiration**: 5-minute automatic cache invalidation
- **Memory Bounds**: Maximum 1000 cached queries
- **Thread Safety**: Concurrent read/write operations supported
## HTTP Layer Streaming Architecture
### Overview
IronDrop v2.7.0 provides advanced HTTP layer streaming for efficient handling of large file uploads. The system automatically switches between memory-based and disk-based processing based on content size, providing optimal performance and resource utilization.
### RequestBody Architecture
```rust
pub enum RequestBody {
Memory(Vec<u8>), // Small uploads (≤64MB)
File(PathBuf), // Large uploads (>64MB)
}
```
The `RequestBody` enum provides a unified interface for handling HTTP request bodies of varying sizes:
- **Memory Variant**: Stores small uploads directly in memory for fast processing
- **File Variant**: Streams large uploads to temporary files to prevent memory exhaustion
- **Automatic Selection**: Transparent switching based on configurable size threshold
- **Resource Management**: Automatic cleanup of temporary files with error recovery
### Streaming Decision Flow
```
HTTP Request → Content-Length Check → Size Threshold Comparison
│
┌─────────────────────────┼─────────────────────────┐
│ ≤64MB │ >64MB │
▼ ▼ ▼
Memory Processing Disk Streaming Disk Streaming
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Read to Vec<u8> │ │ Create Temp File│ │ Stream to Disk │
│ Fast Processing │ │ Stream Chunks │ │ Memory Efficient│
│ Low Latency │ │ Auto Cleanup │ │ Large File Safe │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │ │
└─────────────────────────┼─────────────────────────┘
▼
Unified Processing
(Multipart Parser)
```
### Performance Characteristics
| <1KB | Memory | ~1KB | None | <1ms |
| 1KB-64MB | Memory | ~Size | None | <10ms |
| 64MB-100MB | Disk Streaming | ~64KB | Sequential| <100ms |
| 100MB-1GB | Disk Streaming | ~64KB | Sequential| <1s |
| 1GB-10GB | Disk Streaming | ~64KB | Sequential| <10s |
### Security and Resource Protection
#### Temporary File Management
- **Secure Creation**: Uses `tempfile::NamedTempFile` for secure temporary file creation
- **Automatic Cleanup**: Files automatically deleted when `RequestBody` is dropped
- **Error Recovery**: Cleanup guaranteed even during error conditions
- **Permission Control**: Temporary files created with restricted permissions
#### Resource Limits
- **Memory Protection**: Prevents memory exhaustion from large uploads
- **Disk Space Monitoring**: Checks available disk space before streaming
- **Concurrent Upload Limits**: Configurable limits on simultaneous uploads
- **Size Validation**: Enforces maximum upload size limits
### Integration with Upload System
The HTTP streaming layer integrates seamlessly with the existing upload infrastructure:
```rust
impl UploadHandler {
pub fn handle_upload(&self, request_body: RequestBody) -> Result<(), AppError> {
match request_body {
RequestBody::Memory(data) => {
// Fast path for small uploads
let cursor = Cursor::new(data);
self.process_multipart(cursor)
}
RequestBody::File(path) => {
// Streaming path for large uploads
let file = File::open(path)?;
self.process_multipart(file)
}
}
}
}
```
### Monitoring and Observability
The streaming system provides comprehensive monitoring capabilities:
- **Upload Metrics**: Track memory vs. disk processing ratios
- **Performance Monitoring**: Measure processing times by upload size
- **Resource Usage**: Monitor temporary file creation and cleanup
- **Error Tracking**: Log streaming failures and recovery actions
### Configuration Options
```rust
pub struct StreamingConfig {
pub memory_threshold: usize, // Default: 64MB
pub chunk_size: usize, // Default: 64KB
pub temp_dir: Option<PathBuf>, // Default: system temp
pub max_concurrent: usize, // Default: 10
}
```
### Testing Infrastructure
Dedicated HTTP streaming tests verify correct behavior:
- **`http_streaming_test.rs`**: Comprehensive streaming functionality tests
- **Size Threshold Testing**: Verify correct mode selection
- **Resource Cleanup Testing**: Ensure temporary files are properly cleaned up
- **Error Condition Testing**: Validate error recovery and resource cleanup
- **Performance Testing**: Measure streaming performance across size ranges
## Security Architecture
### Defense in Depth
1. **Input Validation Layer**
- CLI parameter validation with bounds checking
- HTTP header parsing with malformed request rejection
- Multipart boundary validation and size limits
- Filename sanitization and path traversal prevention
2. **Access Control Layer**
- Optional Basic Authentication with secure credential handling
- Rate limiting (120 requests/minute per IP, configurable)
- Connection limiting (10 concurrent per IP, configurable)
- Extension filtering with glob pattern support
3. **Resource Protection Layer**
- Request timeouts to prevent resource exhaustion
- Memory-efficient streaming for large file operations
- Disk space checking before upload operations
- Thread pool management with configurable limits
4. **Audit and Monitoring Layer**
- Comprehensive request logging with unique IDs
- Performance metrics collection and statistics
- Health check endpoints (`/_irondrop/health`, `/_irondrop/status`)
- Unified monitoring dashboard (`/monitor`, `/_irondrop/monitor?json=1`)
- Error tracking and security event logging
### Security Features by Component
| **CLI** | Input validation, path traversal prevention, size bounds |
| **HTTP Layer** | Header validation, method restrictions, rate limiting |
| **Upload System** | File validation, atomic operations, extension filtering |
| **Multipart Parser** | Boundary validation, size limits, malformed data rejection |
| **Template System** | Path restrictions, variable escaping, static asset control |
| **File System** | Canonicalization, directory traversal prevention |
## Performance Characteristics
### Memory Usage
- **Baseline**: ~3MB + (thread_count × 8KB stack)
- **Template Cache**: In-memory storage for frequently accessed templates
- **Upload Buffer**: HTTP streaming with automatic memory/disk switching
- **Small Uploads (≤64MB)**: Direct memory processing for optimal performance
- **Large Uploads (>64MB)**: Disk streaming with ~64KB memory footprint
- **File Operations**: Configurable chunk sizes (default: 1KB)
### Concurrent Processing
- **Thread Pool**: Custom implementation (default: 8 threads)
- **Upload Handling**: Supports multiple concurrent uploads
- **Rate Limiting**: Per-IP tracking with automatic cleanup
- **Connection Management**: Efficient file descriptor usage
### Request Latency
| Static Assets | <0.5ms | CSS/JS with caching headers |
| Directory Listing | <2ms | Template rendering with file sorting |
| Health Checks | <0.1ms | JSON status endpoints |
| File Downloads | Variable | Depends on file size and network |
| File Uploads | Variable | Includes validation and atomic writing |
| Error Pages | <1ms | Template-based professional pages |
### Scalability Limits
- **File Size**: Up to 10GB uploads supported
- **Concurrent Users**: Limited by thread pool and system resources
- **Directory Size**: Efficient handling of large directories
- **Template Complexity**: Sub-millisecond variable interpolation
## Template System Architecture
### Template Engine Design
The native template engine provides:
- **Variable Interpolation**: `{{VARIABLE}}` syntax with HTML escaping
- **Static Asset Serving**: Organized CSS/JS delivery via `/_irondrop/static/` routes
- **Modular Templates**: Separated concerns (HTML structure, CSS styling, JS behavior)
- **Caching**: In-memory template storage for performance
### Template Organization
Each template module follows consistent patterns:
- **HTML**: Clean semantic structure with accessibility features
- **CSS**: Professional design with CSS custom properties
- **JavaScript**: Progressive enhancement with graceful degradation
### Asset Pipeline
```
Template Request → Template Engine → Variable Interpolation → HTML Response
↓
Static Asset Request → Asset Router → Direct File Serving → CSS/JS Response
```
## Testing Architecture
### Test Organization
- **Unit Tests**: Individual component testing
- **Integration Tests**: End-to-end functionality verification
- **Security Tests**: Boundary and vulnerability testing
- **Performance Tests**: Load and stress testing scenarios
### Test Coverage by Component
| `integration_test.rs` | Core server functionality | 19 |
| `integration_test.rs` | Authentication and security | 6 |
| `upload_integration_test.rs` | Upload system | 29 |
| `multipart_test.rs` | Multipart parser | 7 |
| `debug_upload_test.rs` | Edge cases and debugging | Variable |
| Others | Template system, HTTP handling | 40+ |
### Custom Test Infrastructure
- **Native HTTP Client**: Pure Rust implementation for testing
- **Mock File Systems**: Temporary directories and file operations
- **Concurrent Testing**: Multi-threaded test scenarios
- **Security Validation**: Path traversal and injection testing
## Configuration System
### CLI Configuration
```rust
pub struct Cli {
directory: PathBuf, // Required: directory to serve
listen: String, // Default: "127.0.0.1"
port: u16, // Default: 8080
allowed_extensions: String, // Default: "*.zip,*.txt"
threads: usize, // Default: 8
chunk_size: usize, // Default: 1024
verbose: bool, // Default: false
detailed_logging: bool, // Default: false
username: Option<String>, // Optional: basic auth
password: Option<String>, // Optional: basic auth
enable_upload: bool, // Default: false
max_upload_size: u32, // Default: 10240 (10GB)
upload_dir: Option<PathBuf>, // Optional: custom upload dir
}
```
### Validation Pipeline
1. **Parse-time Validation**: Clap value parsers and constraints
2. **Runtime Validation**: Additional checks during server initialization
3. **Operation Validation**: Per-request validation and security checks
## Error Handling System
### Error Types
```rust
pub enum AppError {
Io(std::io::Error),
InvalidMultipart(String),
InvalidFilename(String),
PayloadTooLarge(u64),
UnsupportedMediaType(String),
InvalidConfiguration(String),
// ... additional error variants
}
```
### Error Propagation
- **Result Types**: Consistent error handling throughout the application
- **Error Context**: Detailed error messages for debugging
- **User-Friendly Messages**: Professional error pages with guidance
- **Logging Integration**: Error events logged for monitoring
## Deployment Considerations
### Single Binary Deployment
- **Zero Dependencies**: Pure Rust implementation
- **Embedded Assets**: Templates compiled into binary
- **Cross-Platform**: Supports Linux, macOS, and Windows
- **Portable**: Self-contained executable with no external dependencies
### Production Hardening
- **Security Defaults**: Safe defaults with explicit feature activation
- **Resource Limits**: Configurable bounds to prevent abuse
- **Monitoring Integration**: Health endpoints for infrastructure monitoring
- **Graceful Degradation**: Error recovery and fallback mechanisms
### Scalability Options
- **Reverse Proxy**: Can be deployed behind nginx/Apache for additional features
- **Load Balancing**: Multiple instances can serve the same content
- **Resource Scaling**: Configurable thread pool and memory limits
- **Container Deployment**: Docker-friendly single binary
## Future Architecture Considerations
### Potential Enhancements
1. **Database Integration**: Optional metadata storage for advanced features
2. **Plugin System**: Extensible architecture for custom functionality
3. **WebSocket Support**: Real-time features like upload progress
4. **Distributed Storage**: Support for cloud storage backends
5. **Advanced Auth**: OAuth2/OIDC integration for enterprise deployments
### Performance Optimizations
1. **HTTP/2 Support**: Enhanced protocol capabilities
2. **Async I/O**: Tokio integration for improved concurrency
3. **Compression**: Built-in gzip/brotli compression
4. **CDN Integration**: Edge caching and global distribution
5. **Database Caching**: Redis integration for session management
This architecture documentation reflects the current state of IronDrop v2.7.0 and serves as a foundation for understanding the system's design principles, implementation details, and operational characteristics.