# Development Plan - September 25, 2024
## Today's Achievements (Sept 24)
- ✅ Added support for multiple JOIN conditions with AND
- ✅ Implemented detailed execution plan phase tracking
- ✅ **Massive WIN**: 6x GROUP BY performance improvement by reusing ArithmeticEvaluator
- ✅ Released v1.54.0 with these improvements
## Tomorrow's Focus Areas
### 1. 🔧 Clean Up Hardcoded Keywords in Parser
**Priority: HIGH**
**File:** `src/sql/recursive_parser.rs`
#### Current Issues:
- Keywords are hardcoded with string comparisons throughout
- Multiple `to_uppercase()` calls creating unnecessary allocations
- Pattern matching like `trimmed.to_uppercase().ends_with(" AND")` (lines 1880-1885)
- Inconsistent case handling
#### Tasks:
- [ ] Audit all hardcoded keyword checks in recursive_parser.rs
- [ ] Create a centralized keyword token system or use existing Token enum better
- [ ] Replace string comparisons with token comparisons where possible
- [ ] Eliminate redundant `to_uppercase()` calls
- [ ] Consider creating a `Keywords` module with constants
#### Example Problem Areas:
```rust
// Current (inefficient):
// Better:
if matches!(last_token, Token::And | Token::Or) {
```
### 2. 🌐 Extract WEB CTE into Dedicated Module
**Priority: HIGH**
**Current Location:** Mixed into general CTE handling
#### Tasks:
- [ ] Create new module: `src/sql/web_cte/mod.rs`
- [ ] Move `WebCTESpec` and related types
- [ ] Extract WEB CTE parsing logic from recursive_parser
- [ ] Extract WEB CTE execution from query_engine
- [ ] Improve error handling for network operations
- [ ] Add retry logic and timeout configuration
- [ ] Create comprehensive tests for WEB CTE
#### Module Structure:
```
src/sql/web_cte/
├── mod.rs # Public API
├── parser.rs # WEB CTE specific parsing
├── executor.rs # HTTP request execution
├── cache.rs # Caching logic
└── tests.rs # Unit tests
```
#### Future Enhancements (if time permits):
- [ ] Support for more protocols (WebSocket, GraphQL)
- [ ] Better authentication mechanisms
- [ ] Streaming support for large responses
- [ ] Parallel fetching for multiple WEB CTEs
### 3. 🚀 Additional Optimization Opportunities
**Priority: MEDIUM** (if time permits)
#### Quick Wins to Look For:
- [ ] Check for similar patterns to ArithmeticEvaluator creation
- [ ] Look for repeated allocations in hot paths
- [ ] Special fast path for simple column references in GROUP BY
- [ ] Cache compiled regular expressions
#### Parser Performance:
- [ ] Reduce temporary string allocations
- [ ] Pre-size vectors where possible
- [ ] Consider string interning for common identifiers
### 4. 📝 Documentation Updates
**Priority: LOW** (can be done alongside)
- [ ] Update CLAUDE.md with new execution plan features
- [ ] Document the WEB CTE module architecture
- [ ] Add performance tuning guide based on today's findings
## Success Metrics
- Parser should have fewer string allocations and `to_uppercase()` calls
- WEB CTE code should be in its own module with clear interfaces
- Tests should continue to pass
- Benchmarks should show no performance regression
## Notes
- Start with keyword cleanup as it's foundational
- WEB CTE extraction is well-defined and good for code organization
- Keep an eye out for more "easy wins" like today's GROUP BY optimization
- Consider creating benchmarks for parser performance
## Questions to Consider
1. Should we create a lexer keyword cache?
2. Should WEB CTEs support connection pooling?
3. Is it worth creating a query optimization framework?
## Code Quality Goals
- Reduce code duplication
- Improve separation of concerns
- Make the codebase more maintainable
- Keep performance improvements from today
---
*Ready to tackle these improvements tomorrow! The keyword cleanup will make the parser cleaner and more efficient, while the WEB CTE extraction will improve modularity.*