1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
/*
** 2020-04-01
**
******************************************************************************
** Public interface used by Cloud Backed SQLite (CBS) database clients.
*/
/*
** OVERVIEW
**
** This module provides a VFS that provides access to databases stored in
** Cloud-Backed-SQLite format in remote cloud containers. The VFS may operate
** in two modes:
**
** 1) Daemonless mode. This mode provides read/write access to remote
** databases.
**
** 2) Daemon mode. This mode provides read-only access to remote databases,
** but allows database clients from multiple processes to share a
** single cache directory. This mode requires that the daemon process
** be started before any VFS objects are created.
**
** In general, the same API is used to interface with a daemon or daemonless
** mode VFS. Please refer to the website documentation for more information.
**
** APPLICATION PROGRAMMING
**
** Creating and destroying VFS objects:
**
** The application creates a Cloud Backed SQLite VFS object using the
** sqlite3_bcvfs_create() API. Parameters to this call specify the name of
** the VFS to create, and the path to a directory that the VFS will use to
** store its various files. The directory should either be completely empty,
** or a directory used previously by a Cloud Backed SQLite VFS, or a directory
** that has a running CBS daemon using it. If the directory has a daemon
* running in it, then the new VFS automatically connects to
** that daemon and runs in daemon mode. Otherwise, if there is no daemon,
** then the created VFS object runs in daemonless mode.
**
** If the specified directory does not have a running CBS daemon, but was
** used previously, then the new VFS object is initialized based on the state
** of the directory - meaning that all containers that were attached when
** the previous VFS was shutdown are automatically reattached.
**
** As well as creating a VFS object that may be used by database clients,
** a successful call to sqlite3_bcvfs_create() returns an opaque handle
** (type sqlite3_bcvfs*) that may be used to configure the VFS, attach
** and detach containers, and other operations described below.
**
** Once an application has finished with it and all database clients have
** disconnected, the application may destroy the CBS VFS object
** and reclaim all associated resources using sqlite3_bcvfs_destroy().
**
** An application may create any number of VFS objects, but usually one
** is sufficient. A single VFS object may be used to access databases
** stored in multiple cloud containers across multiple providers.
**
** Configuring VFS objects:
**
** Once a VFS object has been created, it must be configured. A VFS object
** may be configured or reconfigured at any time, but most applications
** will find it sufficient to configure the object once immediately after
** it has been created.
**
** Almost all applications will need to configure the VFS object with
** an authentication callback using sqlite3_bcvfs_auth_callback(). This
** callback is the only way the authentication data (e.g. an Azure SAS
** string or Google Cloud access token) may be provided to cloud storage
** modules that require it.
**
** Application logging (i.e. for debugging) is configure using
** sqlite3_bcvfs_log_callback().
**
** Various integer parameters may also be configured using
** sqlite3_bcvfs_config(). Notably, the SQLITE_BCV_CACHESIZE parameter
** may be usd with sqlite3_bcvfs_config() to configure the maximum size of
** the local block cache in bytes.
**
** Attaching and detaching containers:
**
** Before databases may be accessed, the cloud container that they reside
** in must be "attached" to the VFS. Containers are attached using the
** sqlite3_bcvfs_attach() API function. To attach a container, the
** application specifies cloud storage system (e.g. "azure" or "google"),
** the user-name used to connect to cloud storage, the name of the cloud
** storage container as used by the remote system, and an alias that will
** be used to refer to the container locally.
**
** Unused containers may be detached from a VFS using sqlite3_bcvfs_detach().
**
** Opening and accessing databases:
**
** Once one or more containers have been attached to the VFS, databases
** contained within them may be opened and accessed by SQLite database
** clients. This is done by specifying a path consisting of the container
** alias (local name) and the database name, as follows:
**
** /ALIAS/DATABASE
**
** and using either an SQLite URI filename or the sqlite3_open_v[23]()
** interface to specify the VFS name.
**
** For example, with all error handling omitted, to create a new VFS
** that uses directory "dirname" to store its files, then to use it
** to access database "mydb.db" in container "mycont" belonging to
** Azure user "myuser":
**
** sqlite3_bcvfs *pFs;
** sqlite3 *db;
**
** sqlite3_bcvfs_create("dirname", "myvfs", &pFs);
** // ... configure VFS ...
** sqlite3_bcvfs_attach(pFs, "azure", "myuser", "mycont", "myalias", 0, 0);
** sqlite3_open_v2("/myalias/mydb.db", &db, SQLITE_OPEN_READWRITE, "myvfs");
**
** Assuming URI filenames are enabled (SQLITE_CONFIG_URI), the following
** may be used in place of the sqlite3_open_v2() call above:
**
** sqlite3_open("file:/myalias/mydb.db?vfs=myvfs", &db);
**
** Poll and Upload operations:
**
** The sqlite3_bcvfs_poll() and sqlite3_bcvfs_upload() and
** APIs are used to perform "poll" and "upload" operations on a specified
** attached container. Also available are the "PRAGMA bcv_poll"
** and "PRAGMA bcv_upload" SQL commands.
**
** Poll operations: When a container is first attached, its "manifest" file
** is downloaded from cloud storage. The manifest file contains the list
** of databases in the container and the list of immutable blocks that
** each database consists of. By default, this manifest is used until the
** container is detached, only being updated according to local "upload"
** operations (see below). A "poll" operation downloads a new copy of the
** manifest from cloud storage, making any changes to the list of
** available databases and their contents visible to local clients.
**
** Upload operations: After a database has been modified locally, using the
** usual SQLite interfaces (i.e. SQL statements), the changes may be uploaded
** to cloud storage. An "upload" operation uploads all locally modified
** databases that are part of the specified container.
**
** When databases modifications are uploaded, or databases are deleted
** from cloud storage containers altogether, old, unused blocks are left
** in cloud storage. The system allows these blocks to linger in case
** other clients are still using them instead of deleting them immediately.
** To actually delete old, unused, blocks from cloud storage, see the
** sqlite3_bcv_cleanup() API (in bcvutil.h).
**
** VIRTUAL TABLE INTERFACE
**
** The sqlite3_bcvfs_register_vtab() API is used to register four read-only
** eponymous virtual tables, "bcv_database", "bcv_container", "bcv_block",
** and "bcv_http_log", and one read-write table, "bcv_kv". If the main
** database of the database handle that these are registered with is open on
** a CBS database, the four read-only tables provide information regarding
** the current state of the VFS object. The "bcv_kv" table provides
** read-write access to a key-value store stored in the cloud container
** that may be useful for advisory locks or other communication between
** remote database clients.
**
** The four read-only virtual tables are available to both daemon and
** daemonless mode VFS clients, but bcv_kv is only available in daemonless
** mode.
**
** The bcv_container table:
**
** The "bcv_container" table contains one row for each container attached
** to the VFS. It has the equivalent of the following schema:
**
** CREATE TABLE bcv_container(
** name TEXT, -- local name (alias) of container
** storage TEXT, -- cloud storage system (e.g. "azure")
** user TEXT, -- cloud storage username
** container TEXT, -- container name in cloud storage
** ncleanup INTEGER -- number of blocks eligible for cleanup
** )
**
** The "ncleanup" column usually contains the number of blocks that will
** be deleted from cloud storage if a cleanup operation (see the
** sqlite3_bcv_cleanup() API)is run on the container. However, if
** errors or client crashes have occurred while uploading changes to
** cloud storage, then there may be extra unused blocks left in the
** cloud storage container. In this case those blocks will be deleted
** by the next cleanup operation, but are not included in the value
** stored in the "ncleanup" column of the "bcv_container" table.
**
** The bcv_database table:
**
** The "bcv_database" table contains one row for each database in each
** attached container. It has the equivalent of the following schema:
**
** CREATE TABLE bcv_database(
** container TEXT, -- local name (alias) of container
** database TEXT, -- name of database
** nblock INTEGER, -- total number of blocks in database
** ncache INTEGER, -- number of blocks in cache
** ndirty INTEGER, -- number of dirty blocks in cache
** walfile BOOLEAN, -- true if transactions in local wal file
** state TEXT -- state of database (see below)
** )
**
** The "state" column usually contains an empty string. There are two
** exceptions:
**
** * If the database has been created using sqlite3_bcvfs_copy() but
** not yet uploaded, then this column contains the text 'copied'.
**
** * If the database has been deleted using sqlite3_bcvfs_delete() but
** the delete operation has not yet uploaded, then this column contains
** the text 'deleted'.
**
** The bcv_block table:
**
** The "bcv_block" table contains one row for each block in each database
** in each attached container. It has the equivalent of the following
** schema:
**
** CREATE TABLE bcv_block(
** container TEXT, -- local name (alias) of container
** database TEXT, -- name of database
** blockno INTEGER, -- block number within database
** blockid BLOB, -- block id (or NULL)
** cache BOOLEAN, -- true if block is in cache
** dirty BOOLEAN -- true if block is dirty
** )
**
** The first three columns, "container", "database" and "blockno", identify
** a block within a specific database. Blocks are numbered starting from
** zero.
**
** The "blockid" column usually contains a unique block identifier. Or, if
** the block in question has been appended to the database since it was
** last uploaded, this column contains a NULL value. The boolean "cache"
** column is true if the block is currently cached locally, and "dirty"
** is true if the block is dirty and a new version needs to be uploaded.
**
** The bcv_http_log table:
**
** The "bcv_http_log" table contains one row for each HTTP request made
** by the VFS or connected daemon. It has the equivalent of the following
** for a schema:
**
** CREATE TABLE bcv_http_log(
** id INTEGER, -- Unique, monotonically increasing id value
** start_time INTEGER, -- Time request was made, as iso-8601
** end_time INTEGER, -- Time reply received, as iso-8601 (or NULL)
** method TEXT, -- "PUT", "GET" etc.
** client TEXT, -- Name of client that caused request
** logmsg TEXT, -- Log message associated with request
** uri TEXT, -- URI of request
** httpcode INTEGER -- HTTP response code (e.g. 200)
** )
**
** For those requests that can be associated with a single SQLite database
** handle, the contents of the "client" column is the client name as
** configured using the "PRAGMA bcv_client" command. For requests associated
** with a prefetch operation, it contains the string 'prefetch'.
**
** To prevent it from consuming an ever-increasing amount of memory, entries
** are automatically removed from the bcv_http_log on a first-in/first-out
** according to the values configured for the SQLITE_BCV_HTTPLOG_TIMEOUT
** and SQLITE_BCV_HTTPLOG_NENTRY parameters. Or, in daemon mode, according
** to the values passed via the --httplogtimeout and --httplognentry
** command-line options.
**
** Including various overheads, each entry may be assumed to consume
** 256 bytes of memory or less. So allowing 4096 entries to accumulate in
** the bcv_http_log table may require up to 1MiB of memory.
**
** The bcv_kv table:
**
** The schema of the "bcv_kv" table is:
**
** CREATE TABLE bcv_kv(
** name TEXT PRIMARY KEY, -- Key value
** value -- Associated payload value
** )
**
** The contents of the bcv_kv table is stored as a separate file (an
** SQLite database) in the cloud container - "bcv_kv.bcv". The table
** is empty when the container is first created.
**
** The bcv_kv table implements transaction-like properties. As follows:
**
** * The first time in a database transaction that the bcv_kv table
** is read or written, the file is downloaded from cloud storage and
** cached for the duration of the transaction. All user queries -
** read and write - are executed against this cached version of the
** bcv_kv table. This ensures that each transaction sees a consistent
** version of the table contents.
**
** * When a read/write transaction is committed, a new version of the
** bcv_kv table data is uploaded to cloud storage. At this point,
** if it is found that the file has been modified within cloud storage
** since it was downloaded at the beginning of the transaction (because
** some other client wrote to it), the commit fails and the transaction
** is rolled back. The extended error code in this case is
** SQLITE_BUSY_SNAPSHOT.
**
** Virtual tables without a database:
**
** Sometimes, it may be convenient to access the virtual table interface
** without having to identify and open a cloud container database. To
** support this, paths of the forms:
**
** /
** /CONTAINER
**
** may be opened with bcvfs VFSs, where CONTAINER is the local name (alias)
** of an attached container. The databases opened by either of these two
** forms of path are empty. Attempting to create any new tables within them
** fails. However, if sqlite3_bcvfs_register_vtab() is called on the
** database handle, they may be used to access the bcvfs virtual table
** interface.
**
** If the path opened is simply "/", then only the "bcv_database" and
** "bcv_container" tables are available. If the path is of the form
** "/CONTAINER", then all three virtual tables are available.
*/
extern "C" __cplusplus
} /* end of the extern "C" block */