frankensearch-storage 0.2.1

FrankenSQLite-backed metadata and embedding job storage for frankensearch
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96

# frankensearch-storage

FrankenSQLite-backed metadata and embedding job storage for frankensearch.

## Overview

This crate owns the persistent storage layer for frankensearch, backed by FrankenSQLite. It manages schema bootstrapping, document metadata persistence, content-hash deduplication, an embedding job queue, search history, bookmarks, index build metadata, and staleness detection. It serves as the bridge between frankensearch's in-memory search pipeline and durable on-disk state.

## Key Types

### Storage and Connection

- `Storage` - main storage handle wrapping a FrankenSQLite connection
- `StorageConfig` - configuration for storage initialization

### Document Management

- `DocumentRecord` - stored document metadata record
- `upsert_document` - insert or update a document with dedup
- `list_document_ids` / `count_documents` - document enumeration and counting
- `EmbeddingStatus` / `StatusCounts` - per-document embedding state tracking

### Content Hashing and Deduplication

- `ContentHasher` - SHA-256 content hashing for change detection
- `DeduplicationDecision` - whether to re-embed or skip a document
- `lookup_content_hash` / `record_content_hash` - hash lookup and persistence

### Job Queue

- `PersistentJobQueue` - durable embedding job queue with claim/complete/fail lifecycle
- `ClaimedJob` / `EnqueueRequest` - job queue request and claim types
- `JobQueueConfig` / `JobQueueMetrics` - queue configuration and telemetry
- `QueueDepth` - current queue depth by status

### Pipeline

- `StorageBackedJobRunner` - orchestrates document ingestion through the embedding pipeline
- `IngestRequest` / `IngestResult` / `IngestAction` - ingestion request/result types
- `PipelineConfig` / `PipelineMetrics` - pipeline configuration and performance metrics
- `EmbeddingVectorSink` / `InMemoryVectorSink` - sinks for produced embedding vectors

### History and Bookmarks

- `record_search` / `list_search_history` - search history recording and retrieval
- `add_bookmark` / `list_bookmarks` / `is_bookmarked` - document bookmarking

### Index Metadata and Staleness

- `IndexMetadata` / `IndexBuildRecord` - index build tracking
- `StalenessCheck` / `StalenessReport` - index freshness detection
- `StorageBackedStaleness` - staleness checking backed by stored metadata

### Schema

- `bootstrap` - creates/migrates the storage schema
- `SCHEMA_VERSION` / `current_version` - schema versioning

## Features

| Feature | Description |
|---------|-------------|
| `fts5` | Enables `Fts5LexicalSearch` adapter using FrankenSQLite FTS5 |

## Usage

```rust
use frankensearch_storage::{Storage, StorageConfig, bootstrap};

// Open or create a storage database
let config = StorageConfig::default();
let storage = Storage::open("/path/to/search.db", config)
    .expect("open storage");

// Bootstrap schema
bootstrap(&storage).expect("bootstrap schema");

// Upsert a document
// upsert_document(&storage, &doc).expect("upsert");
```

## Dependency Graph Position

```
frankensearch-core
  ^
  |
frankensearch-storage
  ^
  |-- frankensearch-fsfs
  |-- frankensearch (root, optional, feature: storage)
```

## License

MIT