# SQL-CLI Refactoring Progress & Next Steps
## Date: 2025-01-14
After 13-hour session - Major architectural improvements completed
## ✅ Completed Today
### 1. DataView as Single Source of Truth
- **Pinned Columns**: Fully implemented in DataView with proper boundaries
- Columns stay on left when scrolling
- Navigation respects pinned boundaries (can't cursor into pinned area)
- Cursor follows columns when moving, even with pinned columns
- ASCII [P] indicator for better compatibility
### 2. Sorting System Overhaul
- **3-State Sorting**: Clean implementation (Ascending → Descending → None)
- Sort state tracked in DataView, not scattered across TUI
- `toggle_sort()` method for proper cycling
- Visual indicators (↑ ↓) in status messages
- Works like Excel - repeated presses cycle through states
### 3. ViewportManager Foundation
- Created abstraction layer between DataView and rendering
- Manages visible window, column widths, caching
- Architecture: DataTable → DataView → ViewportManager → Renderer
- Added to F5 debug output
### 4. Memory Optimization Continued
- Further reduction: 700MB → 335MB (52% reduction)
- Removed cloned data from Buffer
- Using indices instead of data copies
## 📋 Tomorrow's Priority Tasks
### Phase 1: Extract Key Management (CRITICAL - Do First!)
```rust
// Move out of enhanced_tui.rs into dedicated modules:
- key_manager/mod.rs
- chord_detector.rs // Chord detection logic
- key_recorder.rs // Recording/playback
- key_dispatcher.rs // Route keys to handlers
- key_bindings.rs // Configurable bindings
```
**Why First**: This is the biggest remaining mess in the TUI. Once extracted:
- Redux actions can be dispatched from clean key handlers
- TUI becomes purely a view layer
- Testing becomes much easier
- Can add configurable keybindings
### Phase 2: Fix Filtering (After Keys)
- Move filter state into DataView (like we did with sorting)
- Implement filter cycling/toggling
- Track case sensitivity in DataView
- Make filter state part of DataView's internal state
### Phase 3: Final TUI Audit
- Check for remaining hacks/anti-patterns
- Look for any remaining direct state mutations
- Identify what can move to ViewportManager
- Document remaining technical debt
### Phase 4: Begin Redux Implementation
With clean key handling, we can:
- Create central store
- Define action types
- Implement reducers
- Convert all state changes to dispatched actions
## 🚀 Future Optimization Ideas
### Query Parallelization
1. **GPU Acceleration (CUDA)**
- Offload simple branches to GPU cores
- Parallel filtering on large datasets
- Aggregate computations (SUM, AVG, etc.)
2. **Rust Parallel Queries (like C# PLINQ)**
```rust
// Use rayon for parallel iteration
use rayon::prelude::*;
// Query optimizer could split work:
let (left_branch, right_branch) = query.split_at_optimization_point();
let (left_result, right_result) = rayon::join(
|| execute_branch(left_branch),
|| execute_branch(right_branch)
);
```
3. **Multi-threaded Query Execution**
- Identify independent branches in query tree
- Execute branches on different threads
- Merge results efficiently
- Consider using tokio for async execution
### Architecture Vision
```
Query Parser
↓
Query Optimizer (identifies parallelizable branches)
↓
Execution Planner
├── CPU Branch (complex logic)
├── GPU Branch (simple parallel ops)
└── Thread Pool Branch (independent subqueries)
↓
Result Merger
↓
DataView
```
## 🔧 Technical Debt Remaining
1. **In enhanced_tui.rs**:
- Key handling logic (tomorrow's priority)
- Some direct buffer mutations
- Mixed concerns (UI + business logic)
- **Code duplication**: `new_with_csv()` and `new_with_json()` are nearly identical
2. **In DataView**:
- Case sensitivity not tracked for filters
- Could optimize visible_rows updates
3. **In ViewportManager**:
- Column width calculation could be smarter
- Need better caching strategy
## 📝 Notes for Tomorrow
1. Start with key extraction - it's the biggest win
2. Keep changes incremental and testable
3. After key extraction, the Redux implementation should be straightforward
4. Consider creating a `state/` module structure for Redux:
```
state/
store.rs
actions.rs
reducers/
data_reducer.rs
ui_reducer.rs
navigation_reducer.rs
```
## 🔨 Quick Refactors to Consider
### Combine new_with_csv and new_with_json
```rust
enum FileType {
Csv,
Json,
}
pub fn new_with_file(file_path: &str, file_type: FileType) -> Result<Self> {
// Common logic for both CSV and JSON
let datatable = match file_type {
FileType::Csv => load_csv_to_datatable(file_path, &table_name)?,
FileType::Json => load_json_to_datatable(file_path, &table_name)?,
};
// Rest is identical...
}
// Then:
pub fn new_with_csv(csv_path: &str) -> Result<Self> {
Self::new_with_file(csv_path, FileType::Csv)
}
pub fn new_with_json(json_path: &str) -> Result<Self> {
Self::new_with_file(json_path, FileType::Json)
}
```
## 💡 Key Insights from Today
1. **Single Source of Truth**: Having DataView own its state (sort, filter, pinned) makes everything cleaner
2. **Abstraction Layers**: ViewportManager proves the value of proper separation of concerns
3. **Incremental Refactoring**: Each improvement makes the next one easier
4. **Memory Efficiency**: Using indices instead of cloning data has massive impact
5. **Code Duplication**: Even in refactored code, we still find duplication (CSV/JSON loaders)
---
*Remember: The goal is to make the TUI a pure view layer that simply renders state and dispatches actions. We're getting close!*