PicoVolt (PVDB)
PicoVolt is an embedded SQL database with queryable history. 2.2.0 adds native encrypted snapshots and vaults, key/password rotation, verified encrypted backup/restore and hybrid full-text/vector retrieval. It builds on the snapshot readers, bounded writer scheduling and durable commit logs introduced in 2.0.
The source and official packages are available without a license fee under the PicoVolt Public-Source License 1.1. This proprietary license permits commercial application embedding and unchanged registry/mirror distribution; standalone competing database products require a separate agreement. This is source-available, not open source. Earlier Apache grants remain unchanged. No account, activation, telemetry or pre-download acceptance is required.
Read encrypted storage, hybrid search and release evidence. Native vault encryption is not offered in WASM. Independent security/cryptographic review and external application trials have not been completed; automated verification is not such an audit.
Start with the 2.0 guide for every maintained interface for atomic batches, persistence choices, log diagnostics and error recovery. The standalone review prompt defines an independent assessment and external trials deferred beyond the 2.0 release.
2.3 implementation branch: persistent retrieval
This branch prepares 2.3.0, adding optional named persistent BM25/full-text and exact vector indexes. DDL, current-snapshot filtered retrieval, hybrid, transactional maintenance, format-8 persistence and inspection are implemented. Historical/transformed queries preserve correctness through the legacy fallback. Open performs source verification; commits may rewrite affected index envelopes and retain older CAS generations. Read the 2.3 guide and qualification ledger before evaluating this candidate.
This PR does not publish 2.3 packages. The published 2.2 quick start below is preserved rather than presenting an unavailable registry install as verified.
Quick start
The 2.2.0 release is free to download and use within its license. Historical 2.0 documentation below remains useful for unchanged APIs. See the transition policy for continuing Apache rights and support.
use Database;
For other languages, see Install and distribution.
Storage model
The engine decouples query logic from storage representation through a Virtualization Layer Engine (VLE) that shifts between two on-disk shapes:
- Development mode: a
.pv/workspace of mutable, append-only chunk files plus a content-addressed blob store and inspectable manifest. - Production mode: a single contiguous, memory-mappable
.pvdbfile produced bypv_bake().
New records use a slotted row layout for O(1) appends. Applications can run bounded maintenance steps that transpose immutable non-tail pages into an MVCC-preserving columnar layout with packed decimal encoding.
Status
The engine is exercised by Rust unit and integration suites plus doctests and maintained-binding integration tests. CI also enforces formatting and warning-free Clippy builds on Linux and Windows. Shipped changes are tracked in CHANGELOG.md, and future work is tracked in ROADMAP.md.
Module map
| Module | Responsibility |
|---|---|
core/types.rs |
constants, ids, PageType, RecordEnvelope, page and file headers (explicit little-endian codecs) |
core/errors.rs |
unified PvError and ComplianceError |
core/value.rs |
dynamically-typed Value and Row |
storage/page.rs |
slotted row page (O(1) append), chain links, columnar transposition |
storage/cache.rs |
bounded LRU buffer pool (enables larger-than-RAM reads) |
storage/cas.rs |
BLAKE3 content-addressable dedup (memory, dev-files, mmap) |
storage/compress.rs |
Delta-Z, LEB128 varints, dictionary bit-packing |
storage/index.rs |
ordered secondary-index query structure and its persisted value/address encoding (point and range) |
storage/record.rs |
row and record-body serialization with CAS interception |
storage/vle.rs |
dev directory store, owned prod snapshot, streamed reads, bake |
engine/mvcc.rs |
transaction clock and snapshot visibility |
engine/wasm.rs |
sandboxed wasmi extension runtime and the WasmExec backend trait |
engine/interp.rs |
pv-wasm: a from-scratch WASM interpreter (integer subset) |
engine/query.rs |
SQL front-end (CREATE/INSERT/UPDATE/DELETE/DROP, SELECT with projection, AS aliases, DISTINCT, aggregates, GROUP BY/HAVING, WHERE predicates incl. IN/BETWEEN/IS NULL/LIKE, BEFORE, multi-column ORDER BY, LIMIT/OFFSET) |
engine/compliance.rs |
optional, app-driven usage-policy hook (not a license requirement) |
enterprise.rs |
optional, host-owned audit events and honest capability discovery for fleet integrations |
db.rs |
the Database surface that ties it together |
upgrade.rs |
out-of-place format migration with backup and deep verification |
ffi.rs |
C ABI (the capi feature): a panic-safe, C-callable surface wrapping the engine for Go, Python, and C bindings |
Engineering notes
- Explicit little-endian codecs for every on-disk structure, instead of
casting
#[repr(C)]structs, so the file format stays portable and its byte offsets are exact. - Two interchangeable WASM backends. The default is the
wasmiinterpreter (vetted, full WASM). Alongside it,pv-wasm(engine/interp.rs) is a from-scratch interpreter: a hand-written binary decoder and structured-control stack machine covering thei32andi64integer subset. Both implement theWasmExectrait, and a differential test checkspv-wasmagainstwasmito keep it honest. Floats, tables, globals, imports, SIMD, andbr_tableare out of scope forpv-wasmand are rejected rather than mis-run. - Page-backed engine. Tables are append-only chains of hot row pages and
optional packed cold pages, each header linking to the next. Inserts append to
a row tail. Commit cost also includes catalog/index maintenance, retained-log
accounting and the selected durability protocol; it is not uniformly O(1).
Logged 2.0 workspaces persist index definitions to reduce catalog rewrites.
Reads stream through a
bounded buffer pool (
storage/cache.rs), so datasets need not fit in RAM, and opt-in ordered indexes (storage/index.rs) turnWHERE col = valueinto a point lookup and range comparisons such asWHERE col > vinto an ordered scan rather than a full scan. - Selectable durability.
Database::set_durability(Durability::Sync)makes each flushfsyncthe data and commit the manifest atomically (write to a temp file,fsync, then rename). The defaultFastmode uses the OS cache only: fast and durable on a clean exit, but not power-loss-safe. - Concurrent transactions. Native
SharedDatabaseexposes independent snapshot readers and bounded FIFO writers. Logged workspaces sync original pages before overwriting them and publish an ordered physical change stream. Reopening rolls back incomplete writes. Format 6 prevents old binaries from bypassing recovery. Existing 1.x images remain readable and migratable. See the concurrency contract for limits and costs. - Hardened against untrusted input. Opening a
.pvdbor workspace, or running a WASM module, validates manifest hashes (no path traversal), bounds-checks CAS offsets and page chains (no out-of-bounds reads or infinite loops on a crafted file), and meters WASM instructions, memory, and output. The decoders are fuzzed (a cross-platform fuzz-lite test and afuzz/cargo-fuzz crate), andcargo auditcurrently reports no vulnerability failures. Both run in CI. See SECURITY.md.
Build
Rust 1.86 or newer is required.
Examples and benchmarks
# time-travel, CAS dedup, publish (bake)
Install the full CLI with cargo install picovolt --features data-tools, then use pv query,
pv inspect, pv history, pv diff, pv migrate, pv compact, pv import,
pv export, and pv bake.
Parquet/SQLite conversion, query explanations, inspection, resumable baking,
and dataset signing are documented in Data tools.
Compare two MVCC snapshots with:
CSV is the default and identifies each row with an _change column. JSONL uses
{"change":"added|removed","row":{...}}. The diff is deterministic and
duplicate-aware; an update is represented as a removed row followed by an added
row. Copyable Rust, Python, Go, Node, and browser projects are in
starters/; supported adapters are catalogued in
docs/INTEGRATIONS.md.
SQL supports the normal PicoVolt CRUD and schema statements plus projection,
filters, aggregates, grouping, time travel, ordering, and pagination. The
current query surface includes AS/bare table aliases, N-table equality
INNER/LEFT joins, searched CASE WHEN, and the focused LOWER, UPPER,
TRIM, LENGTH, ABS, COALESCE, and NULLIF scalar functions. Schema-light
types, literal defaults, named inserts, and persisted CHECK constraints cover
common adapter DDL. See the precise syntax, examples, type behavior, and
deliberate limits in
docs/SQL.md. Rust callers can cache Database::prepare(...)
templates; C, WebAssembly, JavaScript, Python, and Go expose the same reusable
prepared-statement lifecycle. Callers can use explicit transactions or atomic
Database::transaction(...) closures with both filesystem and in-memory
databases.
Durability is selectable via Database::set_durability (Fast OS-cache default,
or crash-safe Sync with fsync and an atomic manifest).
Native Rust applications can use the 2.0 concurrency surface through
SharedDatabase. It is a cloneable, bounded worker-thread coordinator with
explicit read and write transaction handles, FIFO admission, cooperative
cancellation, and rollback before failed or abandoned writes release the queue.
Explicit readers execute against independent snapshots while one writer updates
the workspace. Logged workspaces use format 7 with commit-sequence anchoring;
use clones of one coordinator for a shared development workspace.
See Shared database concurrency for the contract and
current limits.
Measured results and the methodology are in BENCHMARKS.md. These measurements describe specific workloads and versions, rather than a general throughput guarantee. PicoVolt supports bounded page-cache reads, ordered secondary indexes, MVCC time travel and single-file publication. Shared filesystem transactions use an incremental page journal; the legacy unlogged surface retains full rollback images. Current limits include snapshot copy costs, a focused SQL planner and a single writer. The 2.0 benchmark baseline records candidate measurements and their limits; see the concurrency contract for current resource costs.
Maintenance and migration
Cold-page maintenance is explicit so the host retains ownership of scheduling and threads:
Database::compact_step(max_pages) preserves record addresses, indexes, and
complete MVCC history; it never compacts the mutable tail and leaves a page in
row form when transposition would not save space. Each pass uses the
crash-recoverable transaction protocol: allow bounded journal space for logged
workspaces, or a full rollback image for unlogged workspaces. Baked-image migration is
out-of-place and deeply verified before publication:
The source is never modified and existing destinations are never overwritten. See Migration and compaction.
Install and distribution
| Target | How |
|---|---|
| Rust (crates.io) | cargo add picovolt |
| JavaScript / npm (WebAssembly, browser and Node) | npm install picovolt |
| Python (native wheels) | python -m pip install picovolt |
Go (database/sql and direct API) |
go get github.com/MiniJe/picovolt/bindings/go/v2@v2.3.0, then provide the matching native C ABI library described in bindings/go/ |
| C | Download the matching picovolt-capi-* bundle from the latest release, or run cargo build --release --features capi |
| In-memory (native, no filesystem) | Database::open_memory(), export with bake_to_bytes() |
PicoVolt runs in the browser through its in-memory backend plus an OPFS persistence
wrapper and Web Worker endpoint. Build the WebAssembly
package with wasm-pack build --target bundler --release -- --features wasm, then
import { Db } from "picovolt" and run SQL with db.query(...). See
src/wasm_api.rs for the JavaScript surface.
For native languages, the capi feature builds a shared library exposing a C ABI
(include/picovolt.h, src/ffi.rs). The
bindings/ directory wraps it for Go (cgo) and Python
(ctypes); both return query results as the same JSON shape as the JavaScript
binding. The bindings suit embedded use, not a concurrent server's primary store.
All bindings accept positional ? parameters
(db.query("... WHERE id = ?", [1])), bound as safely-escaped SQL literals. For
a familiar surface, PicoVolt provides a better-sqlite3-inspired JavaScript API
(import Database from "picovolt/sqlite"), a Python DB-API 2.0 module
(import picovolt.dbapi2 as sqlite), and a Go database/sql driver
(bindings/go/pvsql). These are interface adapters, not
drop-in compatibility layers: shared limits include positional ? only and the
intentionally compact SQL grammar. JavaScript and in-memory Rust also expose
rollback-capable transaction wrappers. Native bindings expose the same
transaction lifecycle through the C ABI.
Server mode
An optional HTTP and JSON server reaches the engine over a socket. One dedicated thread owns the database and runs statements serially, while a pool of HTTP worker threads accepts concurrent connections and hands each request to that thread over a channel, so the single-threaded core is unchanged.
Endpoints are POST /v1/query, GET /v1/tx, and GET /v1/health. Loopback use
may omit authentication. A non-loopback bind is refused unless a bearer token is
provided with --token-file or PICOVOLT_SERVER_TOKEN; send it as
Authorization: Bearer .... Query bodies, queues, execution time, rows scanned,
result rows, and response size are bounded. TLS is not built in, so network
deployments still belong behind a TLS-terminating reverse proxy. See
src/bin/server.rs.
The HTTP API is sessionless, so each request is an atomic statement and explicit transaction-control statements are rejected. Applications needing a multi-statement transaction should use an embedded language binding, where the transaction belongs to one database handle.
Applications accepting SQL from users can also call Database::query_with_limits
directly and choose their own scan, result, memory, and deadline budgets.
Extending PicoVolt
There are two extension paths: sandboxed WebAssembly user-defined functions, and native modules built on the public API. Both are documented in docs/EXTENDING.md.
Project
| Roadmap | ROADMAP.md |
| Monetization thesis | docs/MONETIZATION.md |
| Enterprise integration foundation | docs/ENTERPRISE.md |
| Platform and file support | docs/SUPPORT.md |
| Contributing | CONTRIBUTING.md |
| Code of conduct | CODE_OF_CONDUCT.md |
| Changelog | CHANGELOG.md |
| Security policy | SECURITY.md |
License
PicoVolt 2.3.0 is distributed under the PicoVolt Proprietary Lifetime License
and the public-release authorization recorded in legal/PUBLIC-RELEASE.json.
Earlier Apache-2.0 components retain their existing grants; the original license
is preserved here. Third-party licenses and notices
continue to apply (see NOTICE).
The optional compliance module is not a license
requirement. It is an opt-in helper for applications that want to enforce their
own usage policy. It does not implement license activation or telemetry.