vibesql-executor 0.2.0

# Columnar Execution Phase 3 - Implementation Status

## Goal
Integrate columnar operations with SIMD acceleration to achieve 6-10x performance improvement on TPC-H queries.

## Completed Work

### 1. Architecture Analysis ✅
- Analyzed existing executor (functional model, not operator tree)
- Found existing columnar module with 8-10x proven speedup
- Identified integration points for SIMD operations

### 2. SIMD Operations Review ✅
- PR #2383 provides SIMD operations via `wide` crate:
  - `simd_add_f64/i64`, `simd_sub_f64/i64`, `simd_mul_f64/i64`, `simd_div_f64/i64`
  - `simd_gt_f64/i64`, `simd_lt_f64/i64`, `simd_eq_f64/i64`, etc.
  - `simd_sum_f64/i64`, `simd_avg_f64/i64`, `simd_min/max_f64/i64`
- All operations work on `&[i64]` and `&[f64]` slices directly
- 4-6x speedup for arithmetic, 4-8x for filtering, 5-10x for aggregation

### 3. Design Decision ✅
**Chosen Approach**: True columnar batches with type-specialized column arrays

**Rationale**:
- Maximizes Rust's compile-time specialization
- Direct compatibility with SIMD operations (`&[i64]`, `&[f64]`)
- Perfect cache locality for vectorized operations
- Zero-copy column views with ownership safety

**Architecture**:
```rust
pub struct ColumnarBatch {
    row_count: usize,
    columns: Vec<ColumnArray>,
}

pub enum ColumnArray {
    Int64(Vec<i64>, Option<Vec<bool>>),    // SIMD-ready
    Float64(Vec<f64>, Option<Vec<bool>>),  // SIMD-ready
    String(Vec<String>, Option<Vec<bool>>),
    Boolean(Vec<u8>, Option<Vec<bool>>),
    Mixed(Vec<SqlValue>),  // Fallback
}
```

**Data Flow**:
```
Vec<Row> → ColumnarBatch → SIMD Filter → SIMD Join → SIMD Aggregate → Vec<Row>
         ↑ convert        ↑ zero-copy &[i64]/&[f64] views      ↑ convert
```

### 4. Implementation In Progress 🔄

**Created**:
- `crates/vibesql-executor/src/select/columnar/batch.rs` (700+ lines)
  - ColumnarBatch struct with column arrays
  - Type-specialized ColumnArray enum
  - Row → ColumnarBatch conversion
  - ColumnarBatch → Row conversion
  - SIMD-ready accessors (`as_i64()`, `as_f64()`)
  - NULL handling with separate bitmasks
  - Comprehensive test suite

**Status**: Compilation errors remain (type mismatches, DataType enum issues)

## Next Steps

### Phase 1: Complete Batch Implementation (1-2 days)
1. Fix compilation errors in `batch.rs`:
   - Resolve DataType enum usages (`DataType::Varchar` vs `DataType::Varchar(None)`)
   - Fix `len()` method signatures for all ColumnArray variants
   - Complete `data_type()` implementation

2. Run and fix unit tests:
   ```bash
   cargo test --package vibesql-executor columnar::batch
   ```

3. Verify round-trip conversion works:
   ```rust
   let batch = ColumnarBatch::from_rows(&rows)?;
   let converted = batch.to_rows()?;
   assert_eq!(rows, converted);
   ```

### Phase 2: Integrate SIMD into Filter (2-3 days)
1. Create `crates/vibesql-executor/src/select/columnar/simd_filter.rs`:
   ```rust
   pub fn simd_filter_i64(
       column: &[i64],
       nulls: Option<&[bool]>,
       predicate: &Predicate,
   ) -> Vec<bool> {
       match predicate {
           Predicate::GreaterThan(val) => simd_gt_i64(column, *val),
           Predicate::LessThan(val) => simd_lt_i64(column, *val),
           // ... other predicates
       }
   }
   ```

2. Update `filter.rs` to use SIMD operations when available:
   ```rust
   if let Some((values, nulls)) = batch.column(idx).and_then(|c| c.as_i64()) {
       let mask = simd_filter_i64(values, nulls, &predicate);
       // Apply mask...
   }
   ```

### Phase 3: Implement Columnar Aggregate (3-4 days)
1. Create `crates/vibesql-executor/src/select/columnar/simd_aggregate.rs`:
   ```rust
   pub fn simd_aggregate_i64(
       column: &[i64],
       nulls: Option<&[bool]>,
       filter: Option<&[bool]>,
       op: AggregateOp,
   ) -> SqlValue {
       match op {
           AggregateOp::Sum => {
               let sum = simd_sum_i64(column);
               SqlValue::Integer(sum)
           }
           AggregateOp::Avg => {
               let avg = simd_avg_i64(column)?;
               SqlValue::Double(avg)
           }
           // ... other aggregates
       }
   }
   ```

2. Add GROUP BY support with hash table:
   ```rust
   pub fn columnar_group_by(
       batch: &ColumnarBatch,
       group_cols: &[usize],
       agg_cols: &[(usize, AggregateOp)],
   ) -> Result<ColumnarBatch, ExecutorError>
   ```

### Phase 4: Implement Columnar HashJoin (4-5 days)
1. Create `crates/vibesql-executor/src/select/columnar/simd_join.rs`:
   ```rust
   pub fn columnar_hash_join(
       left: &ColumnarBatch,
       right: &ColumnarBatch,
       left_key: usize,
       right_key: usize,
   ) -> Result<ColumnarBatch, ExecutorError> {
       // Build hash table from right
       // Probe with left using SIMD equality
       // Materialize matched rows
   }
   ```

2. Use SIMD for join key comparison:
   ```rust
   // For i64 join keys
   if let (Some(left_i64), Some(right_i64)) =
       (left.column(left_key).as_i64(), right.column(right_key).as_i64()) {
       // Use simd_eq_i64 for batch equality checks
   }
   ```

### Phase 5: Integration & Testing (2-3 days)
1. Add columnar path detection in `SelectExecutor`:
   ```rust
   fn should_use_columnar(&self, stmt: &SelectStmt) -> bool {
       // Check if query is columnar-compatible:
       // - Simple predicates (=, <, >, BETWEEN)
       // - Numeric/date columns
       // - Aggregates or JOINs on large tables
   }
   ```

2. Add TPC-H integration tests:
   ```rust
   #[test]
   fn test_tpch_q1_columnar() {
       let result = execute_query("SELECT ...");  // TPC-H Q1
       assert!(result.len() > 0);
       // Verify correctness
   }
   ```

3. Run benchmarks:
   ```bash
   cargo bench --bench tpch_benchmark -- q1 q3 q6
   ```

### Phase 6: Performance Tuning (1-2 days)
1. Optimize batch sizes (test 1K, 4K, 16K, 64K rows)
2. Add prefetching hints for large scans
3. Tune hash table sizes for joins
4. Profile hot paths and optimize

## Performance Targets (SF 0.01)

| Query | Current | Target | Status |
|-------|---------|--------|--------|
| Q1    | ~600ms  | <100ms | Not tested |
| Q3    | 724ms   | <100ms | Not tested |
| Q6    | ~50ms   | <10ms  | Not tested |

## Dependencies

**Blocked by**: PR #2383 (SIMD operations) must be merged first
- Status: Has `loom:review-requested` label
- Once merged, rebase this branch

## Files Modified/Created

- `crates/vibesql-executor/src/select/columnar/batch.rs` (new, 700+ lines)
- `crates/vibesql-executor/src/select/columnar/mod.rs` (updated, added exports)

## Estimated Completion

- **Phase 1 (Batch)**: 1-2 days
- **Phase 2 (Filter)**: 2-3 days
- **Phase 3 (Aggregate)**: 3-4 days
- **Phase 4 (Join)**: 4-5 days
- **Phase 5 (Testing)**: 2-3 days
- **Phase 6 (Tuning)**: 1-2 days

**Total**: 13-19 days (2.5-4 weeks)

## Notes

- Rust's type system is perfect for this - zero-cost abstractions everywhere
- SIMD operations are straightforward once columns are properly typed
- Main complexity is in maintaining compatibility with row-based execution
- Performance gains should be significant (6-10x target is achievable)