# SQL CLI Project
Vim-like terminal SQL editor with in-memory query engine for ultra-fast navigation and data exploration. Built in Rust using ratatui.
## Core Philosophy
- **Vim-inspired**: Modal editing, vim keybindings (hjkl navigation, i/a/A insert modes)
- **In-memory queries**: All data loaded to memory for instant filtering/sorting
- **Fast navigation**: Optimized for keyboard-only workflow with minimal latency
- **Power user focused**: Built for rapid data exploration without mouse
## Tech Stack
- **Language**: Rust 1.26.2
- **TUI Framework**: ratatui + crossterm (high-performance terminal UI)
- **SQL Parser**: Custom recursive descent parser with AST (in-memory evaluation)
- **Data Sources**: CSV, JSON files, REST API (cached in memory)
## Build & Test Commands
```bash
# Build
cargo build --release
# Run Rust tests
cargo test
cargo test --test data_view_trades_test # Important DataView tests
# Run Python tests (IMPORTANT: Run after parser/SQL engine changes)
./run_python_tests.sh # Automatically discovers and runs all tests in tests/python_tests/
./run_all_tests.sh # Runs both Rust and Python tests
# IMPORTANT: Always run before committing!
cargo fmt # Required - formats all code to project standards
# Lint
cargo clippy
# Run application
./target/release/sql-cli <file.csv>
./target/release/sql-cli --enhanced <file.json>
# Test SQL queries non-interactively (use for testing new features)
./target/release/sql-cli data/test_simple_strings.csv -q "SELECT * FROM test" -o csv
./target/release/sql-cli data/test_simple_strings.csv -q "SELECT * FROM test" --query-plan # Shows AST
```
## Testing Guidelines (CRITICAL)
### When to Add Python Tests
**ALWAYS add Python tests when:**
- Adding new SQL functions or operators
- Modifying the parser (recursive_parser.rs)
- Adding new string/math methods
- Changing WHERE clause evaluation
- Implementing new SQL features (GROUP BY, aggregates, etc.)
### How to Test SQL Engine Changes
1. **Use non-interactive mode for quick testing:**
```bash
./target/release/sql-cli data/test_simple_strings.csv -q "YOUR_QUERY" -o csv
./target/release/sql-cli data/test_simple_strings.csv -q "YOUR_QUERY" --query-plan
```
2. **Add tests to Python test suite:**
- String methods: `tests/test_string_methods_comprehensive.py`
- General SQL: `tests/test_sql_engine_pytest.py`
- Cross-reference results with pandas for validation
3. **Run Python tests after changes:**
```bash
uv run pytest tests/test_sql_engine_pytest.py -v ./run_python_tests.sh ```
### Test Data Files
- `data/test_simple_strings.csv` - String operations, text methods
- `data/test_simple_math.csv` - Arithmetic, math functions
- Generate new test data in `scripts/` if needed
## Agents (IMPORTANT: Always delegate to these specialized agents)
### rust-build-fixer
**ALWAYS delegate to this agent when:**
- Any `cargo build` or `cargo build --release` fails
- User reports compilation errors (e.g., "I'm getting an error...")
- After writing new Rust code (proactively check compilation)
- Formatting issues reported (cargo fmt failures)
- Type errors, borrow checker issues, or any Rust compilation problems
### rust-test-failure-investigator
**ALWAYS delegate to this agent when:**
- `cargo test` reports ANY failures
- User mentions test failures (e.g., "tests are failing", "test broken")
- After implementing features that might affect tests
- CI pipeline test failures are reported
- Integration test failures occur
### debug-analyzer
**ALWAYS delegate to this agent when:**
- User provides F5 debug output
- Large debug dumps need analysis
- State inconsistency issues in TUI
- Performance bottlenecks need investigation
**CRITICAL**: Do NOT try to fix compilation errors or test failures yourself. ALWAYS delegate to the appropriate agent immediately.
## docs
- place all docs in the docs folder
## Project Structure
- `src/ui/enhanced_tui.rs` - Main TUI interface (key handling to be migrated)
- `src/app_state_container.rs` - Central state management
- `src/data/data_view.rs` - DataView with column operations
- `src/handlers/` - Event handlers (migration in progress)
- `src/action.rs` - Action system for state updates
- `src/sql/` - SQL parsing and AST evaluation
- `integration_tests/` - Integration test suite
- `integration_tests/test_scripts/` - Shell script test suite
## Current Work: Key Handler Migration
Migrating key handling from TUI main loop to dedicated action system. See KEY_MIGRATION_STATUS.md for details.
**Branch**: key_migration_v2 (based on tui_widgets_v1)
**Recently Fixed**:
- Column sorting with pinned columns
- Unified visible_columns architecture
- Key history display (10 keys max, 2s fade)
## Vim-like Features
- **Modal editing**: Insert (i), Append (a/A), Command mode
- **Vim navigation**: hjkl for movement, g/G for top/bottom
- **Fast column ops**: Pin (p), Hide (H), Sort (s) - single keystrokes
- **Search modes**: `/` for column search, `?` for data search, n/N to navigate
- **Visual feedback**: Key history display, mode indicators
## Performance Features
- **In-memory operations**: All queries run on cached data
- **Virtual scrolling**: Handle 100K+ rows smoothly
- **Instant filtering**: Fuzzy search, regex, SQL WHERE - all sub-second
- **Zero-latency navigation**: Optimized keyboard response
- **Smart caching**: Query results cached for instant re-filtering
## Performance Targets
- 10K-100K rows: Interactive queries (50-200ms)
- Complex queries on 100K rows: ~100-200ms
- Memory: ~50MB for 100K rows
## Testing Scripts
```bash
./integration_tests/test_scripts/test_column_ops.sh
./integration_tests/test_scripts/test_sort_cycles.sh
./integration_tests/test_scripts/test_tui_sort_fix.sh
```
## Important Notes
- **ALWAYS run `cargo fmt` before committing** - This is required for all commits
- Windows compatibility required before merge
- Direct DataView state manipulation being removed
- Action system handles all state changes
- F5 shows debug view with internal state
## Documentation
Extensive docs in `docs/` folder covering architecture, refactoring plans, and feature designs.