skybase 0.1.11

The Open-Source Backend & Developer Platform for the AT Protocol (Firebase for ATProto, powered by skyauth)
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
# πŸ“„ Product Requirements Document (PRD)

# `skybase`
### The Turn-Key Micro-AppView Engine & Developer Platform for the AT Protocol ("Firebase-Like DX for ATProto")

---

## ⚑ Executive 1-Pager (The TL;DR)

| Dimension | Specification |
| :--- | :--- |
| **Product** | `skybase`: An open-source, single-binary Micro-AppView engine, local dev sandbox, and developer platform for the AT Protocol (ATProto). |
| **The Core Problem** | In ATProto, user data lives in sovereign Personal Data Servers (PDSs). While single-user authentication and writes are reasonably supported by official client libraries, **multi-user querying is completely unsolved**. An app cannot query 10,000 remote PDSs to render a feed, leaderboard, or search index. Developers are forced to hand-roll custom firehose consumers, backfill pipelines, and database schemas. |
| **The Strategic Wedge** | **The Single-Binary Micro-AppView**: Download one static binary (`./skybase`). Point it at your collection NSID (`com.myapp.review`). It continuously ingests Jetstream, stores records in an embedded SQLite WAL database with JSON1 virtual columns and FTS5 full-text search, and serves an instant REST & WebSocket live-query API. |
| **Three Operating Topologies** | **Topology A (Client-Sovereign / Zero Custody - Default)**: Browser/mobile app authenticates and writes directly to user PDS via `@atproto/oauth-client-browser`; Skybase runs as a zero-custody read indexer (holds 0 credentials).<br/>**Topology B (Daemon-Managed / Bots)**: Server-side daemons hold DPoP sessions via `skyauth` with AES-256-GCM token encryption at rest.<br/>**Topology C (Token-Mediated Session Proxy)**: Keeps sessions alive indefinitely (>2 weeks) for static frontends hosted on GitHub Pages, Wisp, or Tangled. |
| **Developer Experience** | Firestore-like developer ergonomics via `@skybase/client` TypeScript SDK:<br/>`skybase.collection('com.myapp.review').where('rating', '>=', 4).orderBy('date').onSnapshot(setReviews)`.<br/>No Rust toolchain required for frontend developers. |
| **Immediate Focus** | **Phase 1.5: The Thin Vertical Slice**: Prove the end-to-end loop (DPoP Login β†’ PDS Write β†’ Jetstream Ingest β†’ SQLite Upsert β†’ Live Query) on a single collection with Criterion benchmarks before broad expansion. |

---

## 1. Problem Statement & Ecosystem Context

### 1.1 The Decentralized Application Dilemma
In traditional Web2 architectures (Google Firebase, Supabase), backend development is straightforward: developers write to and read from a single centralized database.

In the **AT Protocol (ATProto)**, this mental model is completely inverted:
1. **User Data Sovereignty**: Every record (post, like, review, game move) resides in the user's personal repository (a signed Merkle Search Tree / MST) hosted on their sovereign **Personal Data Server (PDS)**. Applications do not own user data.
2. **The Micro-AppView Requirement**: Because user records are distributed across thousands of independent remote PDSs, an app cannot query them in real time. The application **must run an AppView** β€” an ingestion and indexing engine that consumes the network firehose, filters relevant records, and maintains a queryable aggregated database.
3. **The Undifferentiated Burden**: Hand-rolling an AppView requires managing WebSocket connections, CAR backfill pipelines, Jetstream sequence cursors, schema deserialization, database indexing, and WebSocket live-push diffs. This is where almost all ATProto app attempts die.

### 1.2 Ecosystem Voices: What Developers Actually Need
As highlighted in the Bluesky ecosystem discussion between developers:

> **garrison (`@garrison.corporate.fm`)**:
> *"the highest-leverage thing we can do to grow atproto right now is make it easier for devs to build apps on... 'firebase for atproto' is a good target... hosted components plus a friendlier database api, like firebase tried to be."*
>
> **Bailey Townsend (`@pds.dad`)**:
> *"Ah yes, an open source token mediated backend so oauth sessions can last longer than 2 weeks and devs can host their projects on wisp, tangled, or github pages."*

This feedback crystallizes the two unserved requirements:
1. **The "Friendlier Database API" + Ingestion Engine**: A query layer that abstracts away CAR sync, MST structures, and Jetstream byte streams into clean `.where().orderBy().limit()` and `.onSnapshot()` primitives.
2. **The Token-Mediated Backend for Static Sites**: A session proxy allowing developers hosting static apps on GitHub Pages, Wisp, or Tangled to maintain persistent OAuth sessions (>2 weeks) without browser storage eviction killing user logins.

### 1.3 Prior Art & Landscape Differentiation ("Why Not X?")

| Existing Tool | What It Does | Why It Is Insufficient / Where `skybase` Wins |
| :--- | :--- | :--- |
| **`@atproto/api`** | Official TypeScript XRPC client. | Only speaks to **one user's PDS at a time**. Has no database, cannot perform cross-user queries, and cannot build an aggregated feed or search index. |
| **`@atproto/oauth-client-*`**| Official TypeScript OAuth client. | Handles login flows, but provides zero data storage, firehose indexing, or query infrastructure. |
| **Bluesky's `Tap`** | Internal Go tool for filtering the raw firehose with backfill. | A low-level streaming pipe (event bus). It does **not** give you a queryable database, REST API, SQLite storage, or WebSocket push subscriptions. |
| **`skyware/jetstream`** | Node.js WebSocket client for Jetstream. | A raw WebSocket listener. You still have to hand-roll SQLite schemas, JSON parsing, cursor tracking, deduplication, and query endpoints. |
| **`HappyView`** | Lexicon-driven AppView server in Rust. | Focuses on runtime schema reflection and Lua scripting. Uses `atrium-oauth`. It is an AppView server, not an ergonomic BaaS/developer SDK. |
| **`skybase`** | **Complete Micro-AppView in one binary.** | Combines Jetstream ingestion + historical PDS backfill + SQLite WAL storage + JSON1 virtual indexes + FTS5 full-text search + REST/WebSocket live queries in a single prebuilt binary with zero configuration. |

### 1.4 Project Identity & Namespace Clarity
- **Disambiguation**: `skybase` is an independent open-source developer framework and Micro-AppView engine for the AT Protocol. It has no relationship or affiliation with `skybase.ai` (an enterprise Kubernetes/cloud security compliance product).
- **Public Namespaces**:
  - Rust Crate: `skybase` on crates.io
  - TypeScript SDK: `@skybase/client` and `@skybase/react` on npm
  - GitHub: `https://github.com/mike10010100/skybase`

---

## 2. Core Vision & Strategic Principles

### 2.1 The "Engine vs. Interface" Mandate (Avoiding the Language Trap)
- **The Market Reality**: 85–90% of ATProto application developers write TypeScript/JavaScript, Swift, Kotlin, or Dart. Only 10–15% write Rust or Go.
- **The PocketBase / Supabase / Meilisearch Playbook**:
  - **PocketBase** is written in Go, but 90% of its users write JavaScript and Dart.
  - **Supabase** is written in Elixir, Go, and C, but developers consume it via TypeScript.
  - **Meilisearch** is written in Rust, but developers interact via npm libraries.
- **Rust is an invisible superpower for the engine, not a barrier for the user**:
  - Sustains thousands of Jetstream events/sec with minimal memory (<100MB RAM design target) and zero garbage collection pauses.
  - Single zero-dependency static executable (`./skybase`). No Node.js runtime conflicts, no native node-gyp build failures, no mandatory Docker.
  - Pure Safe Rust (`#![forbid(unsafe_code)]`) with zero memory corruption, leveraging [`skyauth`]'s formally verified DPoP, PKCE, and SSRF filters.
- **The Golden Rule**: Frontend developers consume `skybase` via standard HTTP/WebSocket APIs and the `@skybase/client` TypeScript SDK. **Zero Rust toolchain required.**

### 2.2 The Deployment Spectrum: Answering "Hosted Components"
To accommodate everything from weekend hobbyists on GitHub Pages to production services:

```text
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                               THE DEPLOYMENT SPECTRUM                                  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Tier                 β”‚ Target Audience         β”‚ What Runs Where                       β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Level 0: Local Dev   β”‚ Development / Offline   β”‚ `npx skybase dev` runs on localhost;  β”‚
β”‚                      β”‚                         β”‚ embedded SQLite + mock Jetstream      β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Level 1: 1-Click     β”‚ Independent builders,   β”‚ Single binary on Fly.io / Railway /   β”‚
β”‚ Self-Hosted          β”‚ open-source projects    β”‚ VPS (/mo); 1-click config templates β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Level 2: Managed     β”‚ Static frontend apps    β”‚ Multi-tenant hosted Micro-AppView     β”‚
β”‚ "Skybase Cloud"      β”‚ (GitHub Pages, Tangled, β”‚ cloud; developers register NSIDs and  β”‚
β”‚ [Strategic Horizon]  β”‚ Wisp, Vercel, Netlify)  β”‚ get instant managed API endpoints     β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
```

### 2.3 Strict Engineering Blueprint
`skybase` adheres strictly to the **Production-Grade Rust Best Practices & Architecture Standards** defined in the reference repository ([`rust-best-practices`](https://github.com/mike10010100/rust-best-practices)):
- `#![forbid(unsafe_code)]` crate-wide.
- Strict crate-root lints (`missing_docs`, zero unwrap/expect/panic in production).
- Typed errors (`SkybaseError` via `thiserror`).
- Defensive concurrency: 64-shard `RwLock` partitioning, clock-warp safe math (`saturating_duration_since`), and cancellation token tracking.
- Detailed implementation rules are codified in [`AGENTS.md`](AGENTS.md).

---

## 3. The Three Operating Topologies (Solving Token Custody & Persistence)

To eliminate the conflict between "user data sovereignty" and practical backend operations, `skybase` explicitly supports three distinct operational topologies:

```text
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                                  THREE OPERATING TOPOLOGIES                                     β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Topology                 β”‚ Custodial Footprint      β”‚ Primary Use Case                          β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Topology A: Client-      β”‚ ZERO Token Custody.      β”‚ Default for Web (React/Next.js) & Mobile  β”‚
β”‚ Sovereign (Default)      β”‚ Skybase holds 0 keys.    β”‚ apps. Writes go direct from client to PDS.β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Topology B: Daemon-      β”‚ Managed Custody. Tokens  β”‚ Background daemons, automated bots, and   β”‚
β”‚ Managed (Bots/Daemons)   β”‚ encrypted with AES-256.  β”‚ feed generators requiring offline writes. β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Topology C: Token-       β”‚ Session Proxy. Confidentialβ”‚ Static SPAs on GitHub Pages, Wisp, and  β”‚
β”‚ Mediated Session Broker  β”‚ refresh token rotation.  β”‚ Tangled. Eliminates 2-week session drop.  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
```

### 3.1 Topology A: Client-Sovereign (Zero Skybase Custody β€” Default)
- **How it works**: The user logs in on the client using `@atproto/oauth-client-browser`. DPoP private keys remain in browser IndexedDB or platform keystores. Writes are submitted directly from the client to the user's personal PDS via XRPC.
- **Role of Skybase**: Operates purely as a read/indexing Micro-AppView, consuming public commits from Jetstream.
- **Sovereignty**: **100% absolute.** `skybase` never sees, stores, or proxies user refresh tokens.

### 3.2 Topology B: Daemon-Managed (Backend Automation & Bots)
- **How it works**: Used when a backend bot or feed generator must sign records autonomously without a user in the browser. Powered by [`skyauth`].
- **Honest Security Boundary**: In this mode, the server **is** an OAuth credential custodian.
- **Storage at Rest**: All refresh tokens and private keys stored at rest are encrypted with **AES-256-GCM** under an application master key (`SKYBASE_MASTER_KEY`). In-memory keys implement `zeroize::Zeroize` on drop.

### 3.3 Topology C: Token-Mediated Session Proxy (Static Hosting / GitHub Pages)
- **How it works (Solving Bailey Townsend's Challenge)**: Static single-page applications hosted on GitHub Pages, Wisp, or Tangled rely on public-client OAuth, where browser storage evictions and token rotation kill sessions after ~2 weeks of inactivity.
- **The Solution**: A lightweight deployed Skybase instance acts as a **Confidential OAuth Session Proxy**. Skybase maintains the long-lived DPoP session and rotates refresh tokens continuously in the background, issuing a persistent session token to the static frontend. Users stay logged in indefinitely.

---

## 4. Architectural Blueprint & Data Flow

### 4.1 System Architecture Diagram

```mermaid
flowchart TD
    subgraph ClientLayer ["Client Applications"]
        BrowserApp["Web / Mobile App (Topology A: Zero Custody)"]
        StaticApp["Static Site on GitHub Pages / Wisp (Topology C: Mediated)"]
        BrowserAuth["@atproto/oauth-client-browser<br/>(Ephemeral Keys in IndexedDB)"]
        BrowserApp -.-> BrowserAuth
    end

    subgraph DaemonLayer ["Server Backend (Topology B: Managed Custody)"]
        RustDaemon["Automated Bot / Feed Generator"]
        SkyauthVault["skyauth Token Vault<br/>(AES-256-GCM Encrypted at Rest)"]
        RustDaemon -.-> SkyauthVault
    end

    subgraph SkybaseEngine ["Skybase Micro-AppView Engine (Single Binary or Hosted)"]
        Gateway["REST & WebSocket API Gateway (/api/v1)"]
        SessionMediator["OAuth Session Mediator (Topology C)<br/>(Persistent >2-Week Sessions)"]
        
        subgraph IngestionPillar ["skybase-index (Micro-AppView Core)"]
            JetstreamSub["Jetstream Firehose Consumer (Filtered NSIDs)"]
            BackfillCrawler["PDS Backfill Crawler (com.atproto.sync.getRepo)"]
            CursorTracker["Monotonic Sequence Cursor (Disk WAL)"]
            SqliteStore[("Embedded SQLite WAL<br/>(Canonical records + JSON1 + FTS5)")]
            QueryEngine["High-Speed Query Builder & Indexer"]
        end
        
        subgraph RealtimePillar ["skybase-events"]
            LiveQueryBroker["WebSocket Live-Query Push Broker"]
            ReconciliationEngine["Optimistic State Reconciliation Tracker"]
        end
        
        subgraph ServerWritePillar ["skybase-repo (Optional Daemon Writes)"]
            XrpcClient["XRPC Client (DPoP Signed)"]
        end
    end

    subgraph AtprotoNetwork ["Decentralized ATProto Ecosystem"]
        UserPds["User PDS (Authoritative MST Repository)"]
        JetstreamRelay["Bluesky Jetstream Relay (Global Firehose)"]
        PlcDir["PLC Directory (did:plc & did:web)"]
    end

    %% Topology A (Zero Custody) Flow: Writes go direct to PDS!
    BrowserApp -->|1. Direct Sovereign Write (DPoP)| UserPds
    UserPds -->|2. Broadcast Commit| JetstreamRelay
    JetstreamRelay -->|3. Stream Filtered NSID Commits| JetstreamSub
    JetstreamSub --> SqliteStore
    JetstreamSub --> LiveQueryBroker
    LiveQueryBroker -->|4. Push Realtime Diffs (WebSocket)| BrowserApp
    BrowserApp -->|5. High-Speed Reads & FTS| Gateway
    Gateway --> QueryEngine
    QueryEngine --> SqliteStore

    %% Topology C (Static Mediated Flow): Session kept alive by Skybase
    StaticApp -->|OAuth & Session Delegation| SessionMediator
    SessionMediator -->|PAR & Token Refresh| UserPds
    StaticApp -->|Live Query & Reads| Gateway

    %% Topology B (Server Custodial) Flow: Daemon writes
    RustDaemon --> XrpcClient
    XrpcClient -->|Server-Signed Write| UserPds

    %% Cold Start Backfill
    BackfillCrawler -->|Historical Sync (CAR export)| UserPds
    BackfillCrawler --> SqliteStore
```

### 4.2 The Sovereign Write & Optimistic Reconciliation Flow
ATProto commits take 50ms–500ms to propagate from PDS β†’ Relay β†’ Jetstream β†’ Micro-AppView β†’ Client. Skybase resolves this eventual consistency gap with the **Optimistic State Reconciliation Protocol**:

```mermaid
sequenceDiagram
    autonumber
    actor User as User / Client SDK
    participant PDS as User Sovereign PDS
    participant Relay as Bluesky Relay / Jetstream
    participant SB as Skybase Micro-AppView
    participant DB as SQLite WAL Store

    Note over User, DB: 1. Optimistic Write Initiation (Client Device)
    User->>User: Generate deterministic TID (rkey)
    User->>User: Emit local optimistic state (_syncStatus: "optimistic")
    
    Note over User, PDS: 2. Sovereign Direct Write (Zero Skybase Custody)
    User->>PDS: com.atproto.repo.createRecord (DPoP-signed)
    PDS->>PDS: Commit record to user's MST
    PDS-->>User: Ack with CID (commit hash) & AT-URI
    User->>User: Update state (_syncStatus: "pds_committed")

    Note over PDS, SB: 3. Decentralized Firehose Propagation (50-500ms)
    PDS->>Relay: Publish commit to global firehose
    Relay->>SB: Jetstream WebSocket message (filtered collection)
    
    Note over SB, DB: 4. Embedded Indexing & Realtime Push
    SB->>DB: Upsert into records table & extract JSON1 virtual indexes
    SB->>DB: Commit monotonic sequence cursor
    SB-->>User: WebSocket live-query update (payload + matching TID/CID)
    
    Note over User: 5. Client State Reconciliation
    User->>User: Match incoming TID/CID -> transition to (_syncStatus: "confirmed")
    Note over User: UI remains perfectly stable throughout with zero flicker!
```

---

## 5. Technical Specifications: The Core Engine

### 5.1 Embedded Micro-AppView (`skybase-index`)
- **Filtered Jetstream Ingestion**: Subscribes to Bluesky Jetstream (`wss://jetstream1.us-east.bsky.network/subscribe`) filtering exclusively for target collection NSIDs (e.g. `wantedCollections=com.myapp.review`), drastically minimizing bandwidth and CPU overhead.
- **Dual-Stage Sync (Catch-up & Real-Time)**:
  1. *Catch-up Phase*: On cold start or extended downtime, Jetstream's short replay window (~days) cannot backfill historical data. Skybase crawls PDS repositories via `com.atproto.sync.getRepo` (CAR export) to backfill records.
  2. *Real-Time Phase*: Hands off seamlessly to the live Jetstream WebSocket stream once synchronized.
- **Lexicon-to-SQLite Schema Mapping**:
  - Rather than fragile dynamic DDL (`ALTER TABLE`), Skybase uses a canonical envelope table with SQLite JSON1 virtual columns and full-text search:
  ```sql
  CREATE TABLE records (
      did TEXT NOT NULL,
      collection TEXT NOT NULL,
      rkey TEXT NOT NULL,
      cid TEXT NOT NULL,
      rev TEXT,
      payload TEXT NOT NULL, -- Exact JSON representation of the Lexicon record
      created_at INTEGER NOT NULL,
      indexed_at INTEGER NOT NULL,
      PRIMARY KEY (did, collection, rkey)
  );

  -- Deterministic secondary indexes on extracted JSON fields:
  CREATE INDEX idx_records_rating ON records(collection, json_extract(payload, '$.rating'));

  -- SQLite FTS5 contentless/external-content table for BM25 text ranking:
  CREATE VIRTUAL TABLE records_fts USING fts5(payload, content='records', content_rowid='rowid');
  ```
- **SQLite WAL Optimization**: Write-Ahead Logging (`PRAGMA journal_mode=WAL`), memory-mapped I/O (`PRAGMA mmap_size`), and synchronous normal mode for concurrent multi-reader throughput and sub-millisecond writes.

### 5.2 Realtime Live Queries (`skybase-events`)
- **WebSocket Gateway (`/api/v1/realtime`)**: Clients subscribe to collection streams with filter expressions:
  ```json
  { "action": "subscribe", "collection": "com.myapp.review", "filter": { "rating": { "": 4 } } }
  ```
- **Broadcast Events**: Commits, updates, and deletes are broadcast to connected clients in real time with sequence numbers, rkeys, and CIDs for optimistic reconciliation.
- **Durable Sequence Cursor**: Monotonic Jetstream sequence timestamps are committed to disk WAL, ensuring zero dropped events across daemon restarts.

### 5.3 Local Dev Sandbox & 1-Click Cloud Daemon (`skybase-server`)
- **Local Sandbox (`./skybase dev --mock`)**: Starts a complete self-contained dev environment with a built-in mock Jetstream event generator/replayer and mock PDS receiver, allowing developers to build and test completely offline.
- **Embedded Web Admin Dashboard**: Bundled SPA (`rust-embed`) at `http://localhost:8080/_/admin` for live collection inspection, query benchmarking, and stream health.
- **1-Click Cloud Deployment**: Pre-configured `fly.toml` and `railway.json` templates to deploy Skybase with persistent SQLite storage in 60 seconds (/mo).

### 5.4 Backend Automation & Storage (`skybase-repo`, `skybase-auth`, `skybase-storage`)
- **`skybase-auth`**: Managed session coordinator for background daemons; AES-256-GCM encryption at rest; token-mediated session broker for static frontends.
- **`skybase-repo`**: Server-side XRPC record creation, updates, and atomic batch mutations (`applyWrites`) with DPoP credentials.
- **`skybase-storage`**: PDS blob upload (`uploadBlob`) with SHA-256 CID calculation and MIME magic-byte verification.

---

## 6. Scope Discipline & Tiered Feature Matrix

To eliminate "schedule fantasy" while maintaining uncompromising Rust safety invariants (`#![forbid(unsafe_code)]`, zero panics, typed errors), `skybase` strictly phases its feature surface into three concrete tiers:

### 6.1 Tier 1: The Core Wedge (v0.1 MVP β€” Immediate Implementation Focus)
*The minimal complete product that solves the genuine, unserved ATProto bottleneck.*

| Component | Target Pillar | Scope & Capabilities in v0.1 |
| :--- | :--- | :--- |
| **Embedded Micro-AppView** | `skybase-index` | Targeted Jetstream ingestion for configured NSIDs; embedded SQLite WAL with canonical `records` table; JSON1 virtual indexes; FTS5 full-text search; sub-millisecond multi-user queries. |
| **Historical PDS Backfill** | `skybase-index` | Cold-start & downtime recovery via `com.atproto.sync.getRepo` CAR sync before seamless handoff to live Jetstream stream. |
| **Live Real-Time Queries** | `skybase-events` | WebSocket live-query streaming (`/api/v1/realtime`) with push updates on record commits, updates, and deletes. |
| **Local Dev Sandbox & 1-Click Deploy** | `skybase-server` | Single static binary (`./skybase dev`) with embedded SQLite; `--mock` mode providing offline synthetic Jetstream event replay; pre-configured `fly.toml` & `railway.json` templates for 60-second cloud deployment. |
| **Web Admin Console** | `skybase-server` | Embedded SPA (`rust-embed`) at `/_/admin` for live collection inspection, query benchmarking, and stream health. |
| **Zero-Custody Client SDK** | `@skybase/client` | TypeScript SDK linking browser-side `@atproto/oauth-client-browser` writes directly to PDS with `skybase` live query subscriptions and optimistic state reconciliation. |

### 6.2 Tier 2: Backend Automation & Custodial Operations (v0.2)
*For backend daemons, automated bots, and static frontends (GitHub Pages/Wisp/Tangled) requiring persistent session mediation.*

| Component | Target Pillar | Scope & Capabilities in v0.2 |
| :--- | :--- | :--- |
| **Token-Mediated Session Proxy** | `skybase-auth` | Confidential OAuth session broker for static SPAs (GitHub Pages, Tangled, Wisp) preventing the 2-week session drop caused by browser storage clearing. |
| **Server-Side Repo CRUD** | `skybase-repo` | DPoP-signed XRPC calls (`createRecord`, `putRecord`, `deleteRecord`, `applyWrites`) from daemon to PDS. |
| **Daemon Credential Custody** | `skybase-auth` | Managed OAuth 2.1 lifecycle via `skyauth`; **AES-256-GCM encryption at rest** for refresh tokens; in-memory key zeroization. |
| **Sovereign Blob Management** | `skybase-storage` | PDS `uploadBlob` client, SHA-256 CID computation, and MIME magic-byte validation. |
| **Declarative Event Hooks** | `skybase-events` | In-process asynchronous Rust hooks (`skybase.events().on_create(...)`) with durable sequence cursor persistence. |

### 6.3 Tier 3: Post-v1 & Enterprise Extensions (Strategic Horizon)
*Architected for, but explicitly out of scope for initial release.*

| Extension | Target Capability | Rationale for Deferral |
| :--- | :--- | :--- |
| **Managed "Skybase Cloud"** | Multi-tenant hosted Micro-AppView platform | Hosted multi-tenant service for developers who want zero server management (Supabase Cloud style). |
| **Enterprise Database** | PostgreSQL backend adapter | SQLite WAL easily scales to millions of records on a single node; premature distributed DB complexity. |
| **Embedded Scripting Engine** | QuickJS / Boa JS runtime | HTTP webhooks solve external extensibility cleanly without embedding a JS VM inside the Rust daemon. |
| **Edge CDN Blob Proxy** | Cloudflare R2 / AWS S3 edge cache | Direct PDS blob URLs are sufficient for early stage apps; CDN caching is an optimization, not a blocker. |
| **Multi-Node Clustering** | Raft consensus / distributed indexers | Single-binary simplicity is the primary differentiator against complex enterprise infrastructure. |

---

## 7. Developer Experience: Code Examples

### 7.1 Frontend Developer Experience (TypeScript SDK)

```typescript
import { SkybaseClient } from '@skybase/client';

const skybase = new SkybaseClient({
  endpoint: 'http://localhost:8080', // or https://api.myapp.com
});

// 1. Direct Multi-User Collection Query
const topReviews = await skybase.collection('com.example.book.review')
  .where('rating', '>=', 4)
  .where('genre', '==', 'cyberpunk')
  .orderBy('createdAt', 'desc')
  .limit(20)
  .get();

// 2. Instant Full-Text Search
const searchResults = await skybase.collection('com.example.book.review')
  .search('Hiro Protagonist Snow Crash')
  .limit(10)
  .get();

// 3. Real-Time Live Query Subscription (with Optimistic Reconciliation)
const unsubscribe = skybase.collection('com.example.book.review')
  .where('itemId', '==', 'book:snow-crash')
  .onSnapshot((reviews) => {
    console.log('Live reviews updated:', reviews);
  });

// 4. Sovereign Write directly to User PDS (Zero Skybase Custody)
const newReview = await skybase.collection('com.example.book.review').create({
  title: 'Snow Crash',
  rating: 5,
  reviewText: 'Essential cyberpunk reading.',
  createdAt: new Date().toISOString(),
});
```

### 7.2 Backend & Systems Developer Experience (Rust Crate)

```rust
use skybase::{Skybase, SkybaseConfig};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = SkybaseConfig::new(
        "https://myapp.com/oauth/client-metadata.json",
        "https://myapp.com/oauth/callback",
        "My ATProto Application",
    )
    .with_jetstream_endpoint("wss://jetstream1.us-east.bsky.network/subscribe");

    let skybase = Skybase::new(config)?;

    // Register declarative trigger on indexed collection
    skybase.events().on_create("com.example.book.review", |event| async move {
        println!("Indexed new review from {}: {}", event.did, event.record.title);
        Ok(())
    });

    Ok(())
}
```

---

## 8. Implementation Roadmap & Milestones

### Phase 1: Foundation & Core Configuration (Completed)
- [x] Standalone repository initialization (`#![forbid(unsafe_code)]`, strict lints, typed errors).
- [x] Integrate `skyauth` dependency and implement `SkybaseConfig` with validation.
- [x] Establish architectural blueprint (`PRD.md`), developer quickstart (`README.md`), and formal `AGENTS.md` guidelines.
- [x] Create public GitHub repository (`https://github.com/mike10010100/skybase`).

### Phase 1.5: The Thin Vertical Slice (De-Risking the Core Thesis β€” Completed)
> *Priority Objective: Rather than attempting to build 8 pillars wide, cut a complete end-to-end vertical slice across one single collection (`app.bsky.feed.post` or custom test NSID). Prove the dual-path thesis with running code and hermetic integration tests before expanding.*
- [x] Implement canonical SQLite `records` table with JSON1 extraction and structured query builder (FTS5 search scheduled for Phase 2).
- [x] Build minimal Jetstream WebSocket consumer filtering for the single target collection.
- [x] Build minimal `createRecord` client using `skyauth` DPoP signing.
- [x] Execute hermetic integration test:
  $$\text{DPoP Login} \longrightarrow \text{createRecord (PDS)} \longrightarrow \text{Jetstream Ingest} \longrightarrow \text{SQLite Upsert} \longrightarrow \text{.where().limit() Query} \longrightarrow \text{WebSocket Event}$$
- [x] Verified and passing with 258 integration, adversarial, stress, and unit tests.

### Phase 2: Production Micro-AppView & Local Dev Sandbox (Tier 1 Core Wedge)
- [ ] Implement FTS5 full-text search integration for BM25 text ranking.
- [ ] Implement dual-stage ingestion: historical backfill crawler (`com.atproto.sync.getRepo` / CAR sync) with seamless handoff to live Jetstream stream.
- [ ] Monotonic cursor tracking with durable disk WAL persistence.
- [ ] Hermetic local dev sandbox (`./skybase dev --mock`) with synthetic Jetstream event replay.
- [ ] Axum 0.7 REST and WebSocket live-query gateway (`/api/v1/collections/*`, `/api/v1/realtime`).
- [ ] Embedded single-binary Web Admin console (`rust-embed`) at `http://localhost:8080/_/admin`.
- [ ] 1-Click cloud deployment templates (`fly.toml`, `railway.json`, `Dockerfile`) with persistent SQLite WAL volume storage.

### Phase 3: Zero-Custody Client SDK & Community Feedback Loop
- [ ] Implement `@skybase/client` TypeScript SDK linking browser-side `@atproto/oauth-client-browser` writes to Skybase live queries.
- [ ] Implement Optimistic State Reconciliation Protocol (`isPending`, `isOptimistic`, `isLagging`) in TypeScript client and React hooks (`@skybase/react`).
- [ ] Record a 5-minute video demo: "Login β†’ Write sovereign record to PDS β†’ Instant local AppView live query".
- [ ] Targeted community review: Bluesky/ATProto developer Discord, atproto Discourse, and Bluesky network thread with targeted questions.

### Phase 4: Backend Automation & Custodial Operations (Tier 2 Expansion)
- [ ] Implement `skybase-auth` managed session coordinator for autonomous daemons and bots.
- [ ] Implement **Token-Mediated Session Proxy** for static frontends (GitHub Pages, Tangled, Wisp) to maintain persistent sessions (>2 weeks).
- [ ] Implement **AES-256-GCM encryption at rest** for daemon-held DPoP refresh tokens (`SKYBASE_MASTER_KEY`).
- [ ] Implement `skybase-repo` server-side XRPC writes (`createRecord`, `putRecord`, `deleteRecord`, `applyWrites`).
- [ ] Implement `skybase-storage` PDS blob upload (`uploadBlob`), SHA-256 CID computation, and MIME header validation.
- [ ] Implement declarative in-process Rust event hooks (`skybase.events().on_create(...)`).

### Phase 5: Production Hardening, Compliance & Reference Apps
- [ ] Security audit, mutation testing sweep, and `cargo deny check` policy enforcement.
- [ ] Reference application templates:
  - Decentralized Book Reviews (`com.example.book.review`)
  - Real-time Micro-Chat (`com.example.chat.message`)
  - Public Polling & Leaderboards (`com.example.poll`)
- [ ] Documentation site and npm release of `@skybase/client`.

### Post-v1 Horizon (Tier 3 Extensions)
- [ ] Multi-tenant managed **"Skybase Cloud"** platform for instant hosted components.
- [ ] Pluggable PostgreSQL backend adapter for multi-node deployments.
- [ ] Edge media CDN proxy with Cloudflare R2 / AWS S3 caching.
- [ ] Embedded lightweight JS/WASM scripting runtime for custom triggers.