RusqDoltLite
This project is a fork of Rusqlite using DoltLite as the SQLite backend. It should be possible to use this library just as you would use rusqlite.
Cloud Backed SQLite block-cache VFS
The optional blockcachevfs feature builds Cloud Backed SQLite's VFS into the
bundled DoltLite archive. The complete pinned Cloud Backed SQLite checkout is
vendored under libdoltlite-sys/cloudsqlite and is included in the
libdoltlite-sys crate package, so builds do not download native sources or
require a source-directory environment variable:
The vendored checkout is from Fossil check-in
50e099ad7bf1d12d747f59b0af973d12809887480463fc9893846b0d6ee22e94; the
corresponding upstream source archive has SHA-256
b322811e8ec4224753f2d9309ed0f011d81c9f2540ce81bd7b5a6a9550d7d03a.
The target still needs libcurl and OpenSSL development headers and libraries.
When pkg-config cannot locate them, set
BLOCKCACHEVFS_CURL_INCLUDE_DIR and BLOCKCACHEVFS_OPENSSL_INCLUDE_DIR.
Google storage keeps its default endpoint exactly
https://storage.googleapis.com. For a Google-compatible test service, pass
an HTTP or HTTPS base endpoint; it must not contain & because it is encoded
in the CBS module selector, and CBS trims trailing slashes before adding the
bucket. The XML API remains the default; select the JSON API explicitly with
AttachSpec::google_json(...) or
AttachSpec::google_json_with_endpoint(...). The auth callback's bearer token
is sent to this endpoint, so use test credentials with emulators:
use AttachSpec;
let attach = google_with_endpoint;
let json_attach = google_json_with_endpoint;
The XML selector google?endpoint=<base-url> requires a compatible Google XML
API endpoint. fake-gcs-server is exercised by the opt-in JSON emulator tests
with -scheme http -port 4443; select the JSON API explicitly because the
existing XML default remains unchanged. Production Google storage keeps the
hard-coded storage.googleapis.com base.
S3 uses AWS Signature Version 4 for every request, including custom
S3-compatible endpoints. The default constructor uses virtual-hosted AWS
addressing for ordinary bucket names and path-style addressing through the
regional AWS endpoint for dotted bucket names, which keeps TLS certificate
validation valid. Custom endpoints use path-style addressing. bucket/prefix
containers are isolated to that prefix, and the native module uses ETags for
conditional fetch, upload, and delete operations:
use ;
let attach = s3
.alias;
let local_s3 = s3_with_endpoint;
// Return this value from the VFS auth callback for temporary credentials.
let secret = s3_secret_with_session_token
.expect;
To bootstrap a new remote database, initialize a new container or prefix and upload a valid local SQLite file before attaching it:
use ;
#
initialize_container is a non-destructive create-if-absent operation: an
existing manifest causes an error and remains unchanged. create_database
requires a non-empty, valid SQLite file; after attachment, use upload to
flush changes to that existing remote database.
The S3 authentication callback returns the secret access key. For temporary
credentials, return secret-access-key\nsession-token using the helper above;
the session token is signed as x-amz-security-token and is never put in the
module selector or URL. Enabling Config::CurlVerbose(true) leaves only
non-header libcurl diagnostic text on stderr; raw HTTP headers and payloads are
omitted, excluding raw credential-bearing HTTP headers from verbose output.
The strict Floci emulator at the endpoint above is a convenient local SigV4
test service.
To reuse CBS's own emulator tests, run the shared runner once per backend. It
uses the same vendored checkout, applies this repository's numbered patches to
a disposable copy, and runs the upstream util_api1.test and
util_upload2.test plus bcvfs_poll1.test; it does not duplicate their test
logic, and rewrites only that test's hard-coded container names to unique names
in the disposable copy:
BLOCKCACHEVFS_S3_ENDPOINT=http://127.0.0.1:14567 \
BLOCKCACHEVFS_GOOGLE_JSON_ENDPOINT=http://127.0.0.1:14091 \
The runner requires Tcl, libcurl, and OpenSSL development headers to build CBS's Tcl test binary. It exits with status 77 when an endpoint is not configured.
For a multi-tenant endpoint, pass the remote container as bucket/prefix and
choose a slash-free local alias. Attach, read, write, and upload paths then
remain under that prefix:
use AttachSpec;
let attach = google
.alias;
Orphan cleanup and object listing are separate privileged maintenance operations; the current low-level list URL is bucket-root oriented.
use ;
#
In-process remote server
Enable the remote feature to embed DoltLite's HTTP remote server:
[]
= { = "rusqdoltlite", = "0.40.21", = ["remote"] }
use ;
Authentication
For a single server, DoltLite can terminate TLS and authenticate clients against a directory of public JWK files:
use Duration;
use ;
This native mode is a server-wide allowlist: every key in authorized-keys
can access every database and operation on that listener. It is suitable for a
single-tenant server, but it is not a distributed authorization system.
The DoltLite HTTP client loads a private credential from DOLTLITE_CREDS_DIR
(default ~/.doltlite/creds) and attaches a freshly signed, 30-second bearer
JWT to each HTTPS request. The audience defaults to the remote hostname, so a
generic host at https://remotes.example.com/<db> should validate
remotes.example.com. DOLTLITE_CREDS_KID selects a credential and
DOLT_OVERRIDE_GRPC_JWT_AUDIENCE overrides the audience. Credentials are never
sent on plain HTTP remotes.
For a multi-tenant or serverless host, terminate HTTPS and authenticate at the
public gateway. Treat the token's kid as an untrusted lookup hint, fetch the
public JWK from the host's database, KV store, or identity service, verify the
Ed25519 signature and Dolt JWT claims,
and then authorize the resulting user for the requested database and operation.
Route the request to a loopback-only
RemoteServer owned by the stateful repository shard; do not configure
authKeysDir on that internal listener. A short-lived per-instance public-key
cache can reduce lookups, provided revocation invalidates or versions the cache.
The host's registration endpoint stores only the public JWK and maps its
derived kid to a user; the client's private seed remains on the client. Set
DOLTLITE_LOGIN_URL to that registration page to customize the instructions
printed by dolt_creds_new(); DoltLite does not perform the registration
request itself.
The gateway must preserve the request method, path, binary body, status, and
Content-Length. The protocol endpoints are:
| Access | Requests |
|---|---|
| Read | GET /<db>/root, GET /<db>/chunk/<hash>, GET /<db>/refs, POST /<db>/has-chunks, POST /<db>/get-chunks |
| Write | POST /<db>/chunks, POST /<db>/commit, PUT /<db>/refs, PUT /<db>/refs-if |
Authentication does not make the data plane stateless: the native server opens database files beneath its configured directory. A serverless control plane therefore still needs repository affinity to a persistent shard, container, or volume. A purely stateless function cannot host the native remote server by itself.