substreams-database-change 4.0.0

# Implementation Plan: NumericAddable Trait for add/sub Operations

## Overview

Implement a `NumericAddable` trait that uses the AddAssign pattern to optimize `add()` and `sub()` operations by mutating a BigDecimal directly, reducing allocations from 4 to 2 per operation.

## Current Implementation Issues

**Location:** [src/tables.rs:410-498](src/tables.rs#L410-L498)

The current `add()` and `sub()` methods:
- Accept `<T: ToDatabaseValue>` (any type that converts to string)
- Convert ALL values to BigDecimal for computation
- Create 4 allocations per operation:
  1. Parse existing value → BigDecimal
  2. Parse new value → BigDecimal
  3. Add BigDecimals (intermediate result)
  4. Convert result to string

**Performance bottleneck:** Simple integer counters (common in blockchain data) pay the full BigDecimal allocation cost every time.

## Design: NumericAddable Trait

### Trait Definition

```rust
/// Trait for types that can be added to a BigDecimal in-place
pub trait NumericAddable {
    /// Add self to the target BigDecimal, mutating it directly
    /// This enables efficient accumulation without intermediate allocations
    fn add_assign_to(&self, target: &mut BigDecimal);

    /// Subtract self from the target BigDecimal, mutating it directly
    /// Equivalent to: *target -= self
    fn sub_assign_from(&self, target: &mut BigDecimal);
}
```

### Key Characteristics

- **No base trait required** - stands alone, no dependency on `NumericValue`
- **AddAssign pattern** - mutates existing BigDecimal instead of creating new ones
- **Simple API** - two methods that directly express the operation
- **Type agnostic** - works for integers, BigDecimal, and strings uniformly

## Implementation Details

### For Integral Types (i8, i16, i32, i64, u8, u16, u32, u64)

Use a macro to implement `NumericAddable` for all integral types uniformly:

```rust
macro_rules! impl_numeric_addable_for_integer {
    ($($t:ty),*) => {
        $(
            impl NumericAddable for $t {
                fn add_assign_to(&self, target: &mut BigDecimal) {
                    *target += BigDecimal::from(*self);
                }

                fn sub_assign_from(&self, target: &mut BigDecimal) {
                    *target -= BigDecimal::from(*self);
                }
            }
        )*
    };
}

// Apply to all integer types
impl_numeric_addable_for_integer!(i8, i16, i32, i64, u8, u16, u32, u64);
```

**Performance:** Convert integer → BigDecimal once, add directly to mutable target.

**Note:** Unsigned subtraction can produce negative results (valid for database context).

### For BigDecimal, BigInt and their references

Use a macro to implement `NumericAddable` for BigDecimal/BigInt types:

```rust
macro_rules! impl_numeric_addable_for_bignum {
    ($($t:ty),*) => {
        $(
            impl NumericAddable for $t {
                fn add_assign_to(&self, target: &mut BigDecimal) {
                    *target += self;
                }

                fn sub_assign_from(&self, target: &mut BigDecimal) {
                    *target -= self;
                }
            }
        )*
    };
}

// Apply to BigDecimal and BigInt types (both owned and references)
impl_numeric_addable_for_bignum!(BigDecimal, &BigDecimal, BigInt, &BigInt);
```

**Performance:** Direct BigDecimal operations, no conversion needed.

**Note:** This works because `BigDecimal` implements `AddAssign` and `SubAssign` for both owned and reference types.

### For String and &str

```rust
impl NumericAddable for String {
    fn add_assign_to(&self, target: &mut BigDecimal) {
        let value = BigDecimal::from_str(self)
            .expect("String must contain valid numeric value");
        *target += value;
    }

    fn sub_assign_from(&self, target: &mut BigDecimal) {
        let value = BigDecimal::from_str(self)
            .expect("String must contain valid numeric value");
        *target -= value;
    }
}

impl NumericAddable for &str {
    fn add_assign_to(&self, target: &mut BigDecimal) {
        let value = BigDecimal::from_str(self)
            .expect("&str must contain valid numeric value");
        *target += value;
    }

    fn sub_assign_from(&self, target: &mut BigDecimal) {
        let value = BigDecimal::from_str(self)
            .expect("&str must contain valid numeric value");
        *target -= value;
    }
}
```

**Performance cost:** Parse string to BigDecimal (acceptable per requirements).

### For BigInt (if needed)

```rust
impl NumericAddable for BigInt {
    fn add_assign_to(&self, target: &mut BigDecimal) {
        // Convert BigInt to BigDecimal first
        let value = BigDecimal::from_str(&self.to_string())
            .expect("BigInt should convert to valid BigDecimal");
        *target += value;
    }

    fn sub_assign_from(&self, target: &mut BigDecimal) {
        let value = BigDecimal::from_str(&self.to_string())
            .expect("BigInt should convert to valid BigDecimal");
        *target -= value;
    }
}
```

## Updated Row Methods

### Update add() method signature

**File:** [src/tables.rs:413](src/tables.rs#L413)

```rust
// OLD
pub fn add<T: ToDatabaseValue>(&mut self, name: &str, value: T) -> &mut Self

// NEW
pub fn add<T: NumericAddable>(&mut self, name: &str, value: T) -> &mut Self {
    if self.operation == Operation::Delete {
        panic!("cannot set fields on a delete operation")
    }
    self.accumulate_add(name, value, false);
    self
}
```

### Update sub() method signature

**File:** [src/tables.rs:424](src/tables.rs#L424)

```rust
// OLD
pub fn sub<T: ToDatabaseValue>(&mut self, name: &str, value: T) -> &mut Self

// NEW
pub fn sub<T: NumericAddable>(&mut self, name: &str, value: T) -> &mut Self {
    if self.operation == Operation::Delete {
        panic!("cannot set fields on a delete operation")
    }
    self.accumulate_add(name, value, true);
    self
}
```

### Refactor accumulate_add() implementation

**File:** [src/tables.rs:437-498](src/tables.rs#L437-L498)

**Current complexity:** 62 lines with string manipulation and BigDecimal parsing

**New implementation:**

```rust
fn accumulate_add<T: NumericAddable>(
    &mut self,
    name: &str,
    value: T,
    negate: bool,
) {
    match self.columns.get_mut(name) {
        Some(existing) => {
            // Validate operation compatibility
            match existing.update_op {
                UpdateOp::Unspecified => panic!(
                    "cannot call add/sub() on field '{}' after unspecified - incompatible operations",
                    name
                ),
                UpdateOp::Set | UpdateOp::Add => {
                    // Parse existing value to BigDecimal once
                    let mut target = BigDecimal::from_str(&existing.value)
                        .expect("existing value should be valid BigDecimal");

                    // Mutate BigDecimal directly using trait method
                    if negate {
                        value.sub_assign_from(&mut target);
                    } else {
                        value.add_assign_to(&mut target);
                    }

                    // Convert back to string for storage
                    existing.value = target.to_string();
                    // Keep existing op (Set stays Set, Add stays Add)
                }
                UpdateOp::Max => panic!(
                    "cannot call add/sub() on field '{}' after max() - incompatible operations",
                    name
                ),
                UpdateOp::Min => panic!(
                    "cannot call add/sub() on field '{}' after min() - incompatible operations",
                    name
                ),
                UpdateOp::SetIfNull => panic!(
                    "cannot call add/sub() on field '{}' after set_if_null() - incompatible operations",
                    name
                ),
            }
        }
        None => {
            // No existing value - create new entry starting from 0
            let mut target = BigDecimal::from_str("0").unwrap();

            if negate {
                value.sub_assign_from(&mut target);
            } else {
                value.add_assign_to(&mut target);
            }

            self.columns.insert(
                name.to_string(),
                FieldValue::with_op(target.to_string(), UpdateOp::Add),
            );
        }
    }
}
```

**Improvements:**
- Reduced from ~62 lines to ~45 lines
- Eliminated string manipulation for negation
- Clearer logic flow
- 2 allocations instead of 4

## File Structure

### New files to create

1. **src/numeric/traits.rs** - Trait definition
   ```rust
   use substreams::scalar::BigDecimal;
   use std::str::FromStr;

   pub trait NumericAddable {
       fn add_assign_to(&self, target: &mut BigDecimal);
       fn sub_assign_from(&self, target: &mut BigDecimal);
   }
   ```

2. **src/numeric/impls.rs** - All type implementations
   ```rust
   use substreams::scalar::{BigDecimal, BigInt};
   use std::str::FromStr;
   use crate::numeric::NumericAddable;

   // Macro for implementing NumericAddable on all integer types
   macro_rules! impl_numeric_addable_for_integer {
       ($($t:ty),*) => {
           $(
               impl NumericAddable for $t {
                   fn add_assign_to(&self, target: &mut BigDecimal) {
                       *target += BigDecimal::from(*self);
                   }

                   fn sub_assign_from(&self, target: &mut BigDecimal) {
                       *target -= BigDecimal::from(*self);
                   }
               }
           )*
       };
   }

   // Apply to all integer types
   impl_numeric_addable_for_integer!(i8, i16, i32, i64, u8, u16, u32, u64);

   // Macro for implementing NumericAddable on BigDecimal/BigInt types
   macro_rules! impl_numeric_addable_for_bignum {
       ($($t:ty),*) => {
           $(
               impl NumericAddable for $t {
                   fn add_assign_to(&self, target: &mut BigDecimal) {
                       *target += self;
                   }

                   fn sub_assign_from(&self, target: &mut BigDecimal) {
                       *target -= self;
                   }
               }
           )*
       };
   }

   // Apply to BigDecimal and BigInt types
   impl_numeric_addable_for_bignum!(BigDecimal, &BigDecimal, BigInt, &BigInt);

   // Implementations for String, &str
   // (see detailed examples in "Implementation Details" section)
   ```

3. **src/numeric/mod.rs** - Module exports
   ```rust
   mod traits;
   mod impls;

   pub use traits::NumericAddable;
   ```

4. **src/numeric/tests.rs** - Unit tests
   - Test AddAssign mutation works correctly
   - Test SubAssign mutation works correctly
   - Test type mixing (i64 → String → i64)
   - Test large value precision
   - Test negative results from subtraction

### Files to update

1. **src/lib.rs** - Add module
   ```rust
   pub mod numeric;  // Add this line
   pub mod tables;
   // ... existing modules
   ```

2. **src/tables.rs** - Update imports and methods
   ```rust
   use crate::numeric::NumericAddable;  // Add import

   // Update method signatures (lines ~413, ~424)
   // Update accumulate_add implementation (lines ~437-498)
   ```

## Implementation Steps

1. ✅ **Create module structure**
   - Create `src/numeric/` directory
   - Create `traits.rs`, `impls.rs`, `mod.rs`, `tests.rs`
   - Update `src/lib.rs` to add `pub mod numeric;`

2. ✅ **Define NumericAddable trait**
   - Write trait in `traits.rs`
   - Add necessary imports

3. ✅ **Implement for i64 first** (validate design)
   - Write implementation in `impls.rs`
   - Add basic unit test in `tests.rs`

4. ✅ **Implement for all integer types using macro**
   - Use `impl_numeric_addable_for_integer!` macro
   - Single implementation covers: i8, i16, i32, i64, u8, u16, u32, u64

5. ✅ **Implement for BigDecimal/BigInt types using macro**
   - Use `impl_numeric_addable_for_bignum!` macro
   - Single implementation covers: BigDecimal, &BigDecimal, BigInt, &BigInt

6. ✅ **Implement for String types**
   - String, &str (requires explicit parsing, cannot use macro)

5. ✅ **Update Row methods**
   - Change `add<T>` signature to use `NumericAddable`
   - Change `sub<T>` signature to use `NumericAddable`
   - Refactor `accumulate_add()` to use trait methods

6. ✅ **Run existing tests**
   - All tests in [src/tables.rs:926-1582](src/tables.rs#L926-L1582) must pass
   - Critical tests:
     - `add_plus_add_accumulates_keeps_add_op` (line 1013)
     - `set_plus_add_accumulates_keeps_set_op` (line 1066)
     - `bigdecimal_precision_preserved` (line 1455)

7. ✅ **Add comprehensive new tests**
   - AddAssign behavior verification
   - Type mixing scenarios
   - Edge cases (negative values, zero, large numbers)

## Edge Cases & Testing

### Type Mixing

**Scenario:** User calls `row.add("balance", 100i64)` then `row.add("balance", "0.5")`

**Behavior:**
1. First call: `target = BigDecimal::from_str("0")`, add 100 → stores `"100"`
2. Second call: `target = BigDecimal::from_str("100")`, add "0.5" → stores `"100.5"`
3. Seamless mixing, no special handling needed

### Subtraction with Unsigned Types

**Scenario:** `row.sub("balance", 100u64)` when balance is 50

**Behavior:**
1. Parse "50" as BigDecimal
2. Call `100u64.sub_assign_from(&mut target)` → `target = 50 - 100 = -50`
3. Stores `"-50"` (valid in database context)

### Large Values

**Scenario:** Blockchain wei values (10^18) or larger

**Behavior:**
- All operations go through BigDecimal
- No precision loss, exact arithmetic
- Works identically to current implementation

### Empty Column (first operation)

**Scenario:** First `add()` call on a new column

**Behavior:**
1. No existing entry in `self.columns`
2. Create `target = BigDecimal::from_str("0")`
3. Add value to target
4. Insert new FieldValue with Add operation

## Performance Impact

**Current approach:**
- Parse existing value → BigDecimal (allocation 1)
- Parse new value → BigDecimal (allocation 2)
- Add BigDecimals (allocation 3)
- Convert result to string (allocation 4)

**Optimized approach:**
- Parse existing value → BigDecimal (allocation 1)
- Call `value.add_assign_to(&mut bigdecimal)` - mutates in place
- Convert result to string (allocation 2)

**Result:** ~50% reduction in allocations (4 → 2)

**Real-world impact:** For Substreams processing 1000s of events per block with counter updates:
- Approximately 2x fewer allocations
- Reduced memory pressure
- Faster execution for accumulation-heavy workloads

## Success Criteria

- ✅ All existing tests pass without modification
- ✅ Compile-time type safety (non-numeric types rejected at compile time)
- ✅ Performance improvement: 2 allocations instead of 4
- ✅ Precision maintained for all numeric operations
- ✅ String types accepted with proper validation
- ✅ Type mixing works seamlessly (i64 → String → i64 sequences)
- ✅ Clean API - no breaking changes to public interface
- ✅ Code is simpler and more maintainable than current implementation