midstream 0.2.0

Real-time LLM streaming with inflight analysis
Documentation
# Pattern Detection APIs - Quick Reference

## TL;DR

✅ **Both required APIs are fully implemented and working**

- `find_similar()` - Find similar patterns in time series using DTW
- `detect_pattern()` - Detect if a pattern exists (boolean)

**Location**: `/workspaces/midstream/crates/temporal-compare/src/lib.rs`

**Status**: Production-ready, tested, documented

---

## Quick Start

```rust
use temporal_compare::TemporalComparator;

// Create comparator
let comparator: TemporalComparator<f64> = TemporalComparator::new(100, 1000);

// Example data
let series = vec![1.0, 2.0, 3.0, 4.0, 5.0, 3.0, 4.0, 5.0];
let pattern = vec![3.0, 4.0, 5.0];

// Find all similar patterns
let matches = comparator.find_similar(&series, &pattern, 1.0);
println!("Found {} matches: {:?}", matches.len(), matches);
// Output: Found 2 matches: [(2, 0.0), (5, 0.0)]

// Check if pattern exists
let exists = comparator.detect_pattern(&series, &pattern, 1.0);
println!("Pattern exists: {}", exists);
// Output: Pattern exists: true
```

---

## API Reference

### `find_similar()`

Find all occurrences of a pattern within a time series.

```rust
pub fn find_similar(
    &self,
    series: &[f64],     // Time series to search in
    pattern: &[f64],    // Pattern to find
    threshold: f64      // Maximum DTW distance (lower = stricter)
) -> Vec<(usize, f64)> // Returns: (start_index, distance)
```

**Returns**: Vector of (index, distance) tuples, sorted by distance (best matches first)

**Algorithm**: Dynamic Time Warping (DTW) with sliding window

**Examples**:
```rust
// Exact matches
let series = vec![1.0, 2.0, 3.0, 4.0, 5.0, 3.0, 4.0, 5.0];
let pattern = vec![3.0, 4.0, 5.0];
let matches = comparator.find_similar(&series, &pattern, 0.5);
// Returns: [(2, 0.0), (5, 0.0)]

// Approximate matches
let series = vec![1.0, 2.0, 3.1, 4.2, 4.9];
let pattern = vec![3.0, 4.0, 5.0];
let matches = comparator.find_similar(&series, &pattern, 1.5);
// Returns: [(2, ~0.4)] - approximate match found
```

---

### `detect_pattern()`

Check if a pattern exists anywhere in the time series.

```rust
pub fn detect_pattern(
    &self,
    series: &[f64],   // Time series to search in
    pattern: &[f64],  // Pattern to detect
    threshold: f64    // Maximum DTW distance
) -> bool            // Returns: true if found, false otherwise
```

**Returns**: Boolean indicating pattern presence

**Algorithm**: DTW-based, early-exit on first match

**Examples**:
```rust
// Pattern exists
let series = vec![1.0, 2.0, 3.0, 4.0, 5.0];
let pattern = vec![3.0, 4.0, 5.0];
let found = comparator.detect_pattern(&series, &pattern, 1.0);
// Returns: true

// Pattern doesn't exist
let series = vec![1.0, 2.0, 3.0];
let pattern = vec![10.0, 20.0, 30.0];
let found = comparator.detect_pattern(&series, &pattern, 0.5);
// Returns: false
```

---

## Advanced APIs

### Generic Type Support

Work with any comparable type, not just f64:

```rust
let comparator: TemporalComparator<i32> = TemporalComparator::new(100, 1000);
let haystack = vec![1, 2, 3, 4, 5, 3, 4, 5];
let needle = vec![3, 4, 5];

let matches = comparator.find_similar_generic(&haystack, &needle, 0.1)?;
// Returns: Vec<SimilarityMatch>
```

### Automatic Pattern Discovery

Find recurring patterns without knowing what to look for:

```rust
let comparator: TemporalComparator<char> = TemporalComparator::new(100, 1000);
let sequence = vec!['a', 'b', 'c', 'a', 'b', 'c', 'a', 'b', 'c'];

let patterns = comparator.detect_recurring_patterns(&sequence, 2, 4)?;
// Returns: Vec<Pattern<char>> with detected patterns
```

### Fuzzy Pattern Matching

Group similar pattern variations:

```rust
let comparator: TemporalComparator<i32> = TemporalComparator::new(100, 1000);
let sequence = vec![1, 2, 3, 1, 2, 4, 1, 2, 3];

let patterns = comparator.detect_fuzzy_patterns(&sequence, 3, 3, 0.7)?;
// Groups [1,2,3] and [1,2,4] as similar patterns
```

---

## Understanding Threshold

The `threshold` parameter controls how strict the matching is:

- **Lower threshold** = More strict (exact matches only)
- **Higher threshold** = More lenient (allows variation)

```rust
// Very strict - only exact or near-exact matches
let strict = comparator.find_similar(&series, &pattern, 0.1);

// Moderate - some variation allowed
let moderate = comparator.find_similar(&series, &pattern, 1.0);

// Lenient - significant variation allowed
let lenient = comparator.find_similar(&series, &pattern, 5.0);
```

**Recommendation**: Start with `1.0` and adjust based on results.

---

## Performance Tips

### Caching

All methods use intelligent caching automatically:

```rust
let comparator: TemporalComparator<f64> = TemporalComparator::new(100, 1000);

// First call - computes DTW
let matches1 = comparator.find_similar(&series, &pattern, 1.0);

// Second call - uses cache (much faster)
let matches2 = comparator.find_similar(&series, &pattern, 1.0);

// Check cache performance
let stats = comparator.cache_stats();
println!("Cache hit rate: {:.2}%", stats.hit_rate() * 100.0);
```

### Clear Cache

If memory is a concern:

```rust
comparator.clear_cache();
```

---

## Edge Cases Handled

✅ Empty patterns
```rust
let matches = comparator.find_similar(&series, &[], 1.0);
// Returns: []
```

✅ Pattern longer than series
```rust
let matches = comparator.find_similar(&[1.0, 2.0], &[1.0, 2.0, 3.0, 4.0], 1.0);
// Returns: []
```

✅ Single element patterns
```rust
let matches = comparator.find_similar(&[1.0, 2.0, 3.0, 2.0], &[2.0], 0.5);
// Returns: [(1, 0.0), (3, 0.0)]
```

---

## Testing

### Run Unit Tests
```bash
cd /workspaces/midstream/crates/temporal-compare
cargo test
```

### Run Integration Tests
```bash
cargo test --test temporal_compare_api_test
```

### Run Example
```bash
cargo run --example pattern_detection_demo
```

---

## Documentation

### Generated Docs
```bash
cargo doc --no-deps --open
```

### Documentation Files
- `/workspaces/midstream/docs/temporal_compare_api_verification.md` - Detailed API verification
- `/workspaces/midstream/docs/PATTERN_DETECTION_IMPLEMENTATION.md` - Implementation guide
- `/workspaces/midstream/docs/IMPLEMENTATION_COMPLETE_SUMMARY.md` - Task completion summary

---

## Common Use Cases

### 1. Time Series Anomaly Detection
```rust
// Define normal pattern
let normal_pattern = vec![1.0, 2.0, 3.0, 2.0, 1.0];

// Check if it appears in recent data
let recent_data = vec![5.0, 10.0, 15.0, 20.0, 25.0]; // Anomalous
let is_normal = comparator.detect_pattern(&recent_data, &normal_pattern, 1.0);
// Returns: false (anomaly detected)
```

### 2. Sensor Data Analysis
```rust
// Find all occurrences of a spike pattern
let sensor_data = vec![10.0, 10.0, 50.0, 10.0, 10.0, 50.0, 10.0];
let spike_pattern = vec![10.0, 50.0, 10.0];

let spikes = comparator.find_similar(&sensor_data, &spike_pattern, 2.0);
// Returns: [(1, 0.0), (4, 0.0)] - two spike events
```

### 3. Signal Processing
```rust
// Detect repeating waveforms
let signal = vec![0.0, 1.0, 0.0, -1.0, 0.0, 1.0, 0.0, -1.0];
let waveform = vec![0.0, 1.0, 0.0, -1.0];

let cycles = comparator.find_similar(&signal, &waveform, 0.5);
// Returns: [(0, 0.0), (4, 0.0)] - two complete cycles
```

### 4. Market Data Pattern Recognition
```rust
// Find price patterns (e.g., double top)
let prices = vec![100.0, 110.0, 105.0, 110.0, 100.0];
let double_top = vec![105.0, 110.0, 105.0];

let patterns = comparator.find_similar(&prices, &double_top, 2.0);
// Detects double top pattern
```

---

## Error Handling

All advanced APIs return `Result<T, TemporalError>`:

```rust
use temporal_compare::TemporalError;

match comparator.detect_recurring_patterns(&sequence, 2, 4) {
    Ok(patterns) => {
        println!("Found {} patterns", patterns.len());
    }
    Err(TemporalError::InvalidPatternLength(min, max)) => {
        eprintln!("Invalid lengths: min={}, max={}", min, max);
    }
    Err(e) => {
        eprintln!("Error: {}", e);
    }
}
```

---

## Type Support

Works with any type that implements required traits:

```rust
// f64 (built-in methods)
let comp_f64: TemporalComparator<f64> = TemporalComparator::new(100, 1000);

// Integers
let comp_i32: TemporalComparator<i32> = TemporalComparator::new(100, 1000);

// Characters
let comp_char: TemporalComparator<char> = TemporalComparator::new(100, 1000);

// Custom types (must impl Clone + PartialEq + Debug + Serialize + Hash + Eq)
#[derive(Clone, PartialEq, Debug, Serialize, Hash, Eq)]
struct CustomValue(i32);

let comp_custom: TemporalComparator<CustomValue> = TemporalComparator::new(100, 1000);
```

---

## Summary

| Feature | Status | Location |
|---------|--------|----------|
| `find_similar()` | ✅ Complete | Lines 468-505 |
| `detect_pattern()` | ✅ Complete | Lines 531-536 |
| Generic API | ✅ Complete | Lines 563-633 |
| Recurring patterns | ✅ Complete | Lines 659-740 |
| Fuzzy matching | ✅ Complete | Lines 766-858 |
| DTW algorithm | ✅ Complete | Lines 249-304 |
| Caching | ✅ Complete | Throughout |
| Documentation | ✅ Complete | Doc comments |
| Tests | ✅ Complete | 30+ tests |
| Examples | ✅ Complete | Demo file |

**All pattern detection APIs are production-ready and fully functional.**

---

## Support

- **Source**: `/workspaces/midstream/crates/temporal-compare/src/lib.rs`
- **Tests**: `/workspaces/midstream/crates/temporal-compare/src/lib.rs` (unit tests)
- **Integration Tests**: `/workspaces/midstream/tests/temporal_compare_api_test.rs`
- **Example**: `/workspaces/midstream/examples/pattern_detection_demo.rs`
- **Docs**: `/workspaces/midstream/docs/PATTERN_DETECTION_*.md`