vectorlite 0.1.5

Let me analyze the memory usage patterns in this PR to identify potential inefficiencies.

# Memory Inefficiencies Analysis

Yes, I've identified several significant memory inefficiencies in this PR:

## 🚨 **Critical Memory Issues**

### 1. **Triple Storage in HNSW Index** (Most Critical)
```rust
// In HNSWIndex::add()
let internal_index = self.hnsw.insert(vector.values.clone(), &mut self.searcher);  // 1st copy
self.vectors.insert(vector.id, vector);  // 2nd copy (entire Vector struct)
```

**Problem**: The HNSW index stores vector data in **three places**:
- `self.hnsw` - stores `vector.values.clone()` (just the embedding)
- `self.vectors` - stores the entire `Vector` struct (including text + metadata)
- Plus the original `vector` parameter

**Memory Impact**: For each vector, you're storing:
- Embedding values: `768 * 8 bytes = 6,144 bytes`
- Text: `~100-1000 bytes` (average)
- Metadata: `~200-2000 bytes` (JSON)
- **Total per vector**: ~6,500-9,000 bytes stored **twice** = **13,000-18,000 bytes**

### 2. **Excessive Cloning in Search Results** (High Impact)
```rust
// In both FlatIndex and HNSWIndex search methods
.map(|e| SearchResult {
    id: e.id,
    score: similarity_metric.calculate(&e.values, query),
    text: e.text.clone(),        // String clone
    metadata: e.metadata.clone() // JSON clone
})
```

**Problem**: Every search operation clones all text and metadata for every result, even when only returning top-k results.

**Memory Impact**: For a search returning 10 results from 1000 vectors:
- Clones 10 strings + 10 JSON objects unnecessarily
- Could be 20-30KB of temporary allocations per search

### 3. **String Duplication in Vector Creation** (Medium Impact)
```rust
// In Collection::add_text_with_metadata()
let vector = Vector { 
    id, 
    values: embedding, 
    text: text.to_string(),  // Creates new String
    metadata 
};
```

**Problem**: The input `text: &str` is converted to `String` even though it might be used only once.

## 🔧 **Recommended Fixes**

### Fix 1: Separate Vector Storage from Index Data
```rust
pub struct HNSWIndex {
    hnsw: Hnsw<Euclidean, Vec<f64>, StdRng, ...>,
    // Store only what's needed for search
    vector_data: HashMap<u64, VectorData>,
}

struct VectorData {
    text: String,
    metadata: Option<serde_json::Value>,
}
```

### Fix 2: Lazy Search Results
```rust
pub struct SearchResultRef<'a> {
    pub id: u64,
    pub score: f64,
    pub text: &'a str,           // Reference instead of owned
    pub metadata: Option<&'a serde_json::Value>,
}

// Convert to owned only when needed
impl<'a> SearchResultRef<'a> {
    pub fn to_owned(self) -> SearchResult {
        SearchResult {
            id: self.id,
            score: self.score,
            text: self.text.to_string(),
            metadata: self.metadata.cloned(),
        }
    }
}
```

### Fix 3: String Interning for Common Text
```rust
use std::collections::HashMap;
use std::sync::Arc;

pub struct TextInterner {
    strings: HashMap<Arc<str>, Arc<str>>,
}

impl TextInterner {
    pub fn intern(&mut self, s: &str) -> Arc<str> {
        // Return existing Arc<str> if found, otherwise create new one
    }
}
```

## 📊 **Memory Usage Estimates**

**Current Implementation** (per 1000 vectors):
- HNSW: ~13-18MB (triple storage)
- Flat: ~6.5-9MB (double storage)

**With Fixes** (per 1000 vectors):
- HNSW: ~6.5-9MB (single storage)
- Flat: ~6.5-9MB (single storage)
- **Savings**: ~50% memory reduction

## 🎯 **Priority Recommendations**

1. **High Priority**: Fix HNSW triple storage - biggest memory waste
2. **Medium Priority**: Implement lazy search results with references
3. **Low Priority**: Add string interning for repeated text patterns

The current implementation will use significantly more memory than necessary, especially for large datasets. The fixes would reduce memory usage by approximately 50% while maintaining the same functionality.