worktable 0.8.3

# WARP.md

This file provides guidance to WARP (warp.dev) when working with code in this repository.

## Project Overview

WorkTable is an in-memory storage system with on-disk persistence capabilities, written in Rust. It uses a declarative macro (`worktable!`) to generate type-safe table structures, indexes, and queries at compile time.

## Architecture

### Core Components

**Macro-Generated Code Architecture:**
- `worktable!` macro generates complete table implementations including:
  - `<Name>WorkTable` - Main table struct wrapping the core `WorkTable`
  - `<Name>Row` - Row struct with defined columns
  - `<Name>PrimaryKey` - Type-safe primary key wrapper
  - Custom query methods based on declared queries and indexes

**Internal Storage Architecture:**
- **DataPages**: Manages data as fixed-size pages with atomic operations for lock-free insertions
- **IndexMap**: Primary key index ensuring uniqueness and fast lookups
- **SecondaryIndexes**: Additional indexes for efficient non-primary key searches
- **LockMap**: Supports concurrent operations with row-level locking
- **PersistenceEngine**: Handles async persistence operations (CDC-based)

### Key Architectural Patterns

**Page-Based Storage:**
- Data stored in pages (configurable size, default 4KB) using `rkyv` serialization
- Empty space tracking with lock-free stack for efficient space reuse
- Link-based navigation: `Link { page_id, offset, length }`

**Concurrency Model:**
- Lock-free insertions using atomic operations
- Row-level locking for updates/deletes
- Smart locking for partial row updates (non-overlapping fields)

**Persistence (In Progress):**
- Change Data Capture (CDC) for tracking modifications
- Async persistence with operation queues
- File extensions: `.wt.data` for data, `.wt.idx` for indexes

## Common Development Commands

### Building and Testing

```bash
# Build all workspace members
cargo build

# Build with performance measurements feature
cargo build --features perf_measurements

# Run all tests
cargo test

# Run specific test module
cargo test worktable::base

# Run tests in a specific workspace member
cargo test -p worktable

# Run async tests (many tests use tokio)
cargo test --lib
```

### Development Workflow

```bash
# Check code without building
cargo check

# Run clippy for linting
cargo clippy

# Format code
cargo fmt

# Run a single test file
cargo test --test mod tests::worktable::base

# Run benchmarks (uses performance measurement)
cargo test bench --features perf_measurements

# Build documentation
cargo doc --open
```

### Workspace Structure Commands

```bash
# Work with specific workspace members
cargo build -p worktable_codegen
cargo test -p performance_measurement
cargo run -p examples

# Build everything in workspace
cargo build --workspace

# Clean build artifacts
cargo clean
```

## Working with the Codebase

### Code Generation (Procedural Macros)

The `codegen/` directory contains the procedural macro implementation:
- `worktable/` - Main macro logic for table generation
- `name_generator.rs` - Naming conventions for generated types
- `persist_table.rs` - Persistence-related code generation

When modifying the macro, test with:
```bash
# Test macro expansion
cargo test -p worktable_codegen
# Test generated code works
cargo test worktable
```

### Testing Strategy

**Test Categories:**
- `tests/worktable/base.rs` - Basic CRUD operations
- `tests/worktable/bench.rs` - Performance comparison tests
- `tests/worktable/config.rs` - Configuration and custom types
- `tests/worktable/persistence/` - Persistence functionality
- `tests/worktable/float.rs`, `uuid.rs` - Type-specific tests

**Testing Generated Tables:**
```rust
worktable!(
    name: Test,
    columns: { id: u64 primary_key autoincrement, value: String },
    indexes: { value_idx: value unique },
    queries: { 
        update: { ValueById(value) by id },
        delete: { ByValue() by value }
    }
);
```

### Performance Testing

The codebase includes performance measurement infrastructure:
```bash
# Enable performance measurements
cargo build --features perf_measurements

# Run performance comparison tests
cargo test bench --features perf_measurements
```

### Memory Management

WorkTable focuses on efficient memory usage:
- **Page allocation**: Fixed-size pages minimize fragmentation
- **Link reuse**: Deleted records create reusable links
- **Zero-copy reads**: `rkyv` enables reading without deserialization

### Persistence Development

For persistence features (in progress):
- Operations are queued as CDC events
- Files use custom page format with headers
- Index pages and data pages are managed separately

## Key Files for Understanding

- `src/lib.rs` - Main module exports and prelude
- `src/table/mod.rs` - Core WorkTable struct definition  
- `src/in_memory/pages.rs` - DataPages implementation
- `codegen/src/worktable/` - Macro implementation
- `tests/worktable/base.rs` - Usage examples and basic tests
- `README.md` - Detailed API documentation with examples

## Development Notes

**When adding features:**
- Update both runtime code (`src/`) and code generation (`codegen/`)
- Add comprehensive tests covering the generated API
- Consider memory layout and concurrency implications
- Test with different column types and configurations

**Performance considerations:**
- Page size affects memory usage and cache performance
- Index design impacts query performance
- Lock granularity affects concurrency