excelstream 0.17.0

High-performance streaming Excel & CSV library with S3/GCS cloud support and Parquet conversion - Ultra-low memory usage
Documentation
# S3 Streaming Performance Results

## Test Configuration
- **Library**: excelstream v0.13.0
- **s-zip version**: 0.5.1 (cloud-s3 support)
- **Backend**: AWS S3 (ap-southeast-1)
- **Build**: Release mode with optimizations
- **Columns**: 10 columns per row
- **Implementation**: TRUE streaming (no temp files)

## Memory Usage Results

| Dataset Size | Peak Memory (MB) | Time (s) | Throughput (rows/s) | File Size (est.) |
|--------------|------------------|----------|---------------------|------------------|
| 10K rows     | **15.0 MB**      | 0.91     | 10,951              | ~2 MB            |
| 100K rows    | **23.2 MB**      | 2.20     | 45,375              | ~20 MB           |
| 500K rows    | **34.4 MB**      | 5.31     | 94,142              | ~100 MB          |

## Key Findings

### ✅ Constant Memory Usage
- **10K → 100K rows (10x data)**: Memory only increased 1.5x (15MB → 23MB)
- **100K → 500K rows (5x data)**: Memory only increased 1.5x (23MB → 34MB)
- **Peak memory stays under 35 MB** even for 500K+ rows
- **True streaming confirmed**: No file-size-proportional memory growth

### 🚀 Performance Characteristics
- **Throughput scales with dataset size**: 10K → 94K rows/sec
- **No temp files**: Zero disk usage
- **Multipart upload**: Handled automatically by s-zip
- **Async streaming**: Efficient tokio runtime usage

### 📊 Comparison: Before vs After

#### Old Implementation (v0.3.1 + temp file)
```
❌ Create temp file (~file size disk usage)
❌ Write entire Excel to temp file
❌ Read temp file and upload to S3
❌ Memory: File size + buffer (could be GB for large files)
```

#### New Implementation (v0.5.1 + cloud-s3)
```
✅ Stream directly to S3 (no temp files)
✅ Constant ~30MB memory regardless of file size
✅ S3 multipart upload handled by s-zip
✅ Works in read-only filesystems (Lambda, containers)
```

### 💡 Memory Breakdown
The ~30MB peak memory consists of:
- **~4KB**: XML buffer for row construction
- **~5MB**: S3 multipart upload buffer (per part)
- **~10MB**: AWS SDK client + HTTP connections
- **~15MB**: Tokio runtime + misc overhead

## Example Usage

### Small Dataset (< 10K rows)
```bash
cargo run --example s3_streaming --features cloud-s3 --release
# Peak memory: ~15 MB
```

### Large Dataset (100K+ rows)
```bash
TEST_ROWS=500000 cargo run --example s3_performance_test --features cloud-s3 --release
# Peak memory: ~34 MB (constant!)
```

### Measure Memory Yourself
```bash
/usr/bin/time -v cargo run --example s3_performance_test --features cloud-s3 --release
# Look for "Maximum resident set size"
```

## Conclusion

The new s-zip 0.5.1 cloud-s3 integration achieves:
- **TRUE constant memory usage** (~30-35 MB)
-**Zero disk usage** (no temp files)
-**High throughput** (94K+ rows/sec)
-**Production-ready** for serverless environments

Perfect for:
- AWS Lambda functions
- Docker containers with limited memory
- Read-only filesystems
- Large-scale data processing pipelines