udb 0.4.0

UDB is a public, schema-driven knowledge graph and data broker for applications that need one typed API in front of many data systems.

You describe your domain in normal .proto files, add UDB annotations for storage and security behavior, then run one broker in front of relational databases, object stores, caches, vector stores, document stores, graph stores, analytics systems, and native identity services. Application code calls UDB through gRPC or an SDK; UDB handles routing, metadata, authorization, migrations, CDC, and backend-specific execution.

What UDB Gives You

• One DataBroker API for relational records, objects, vectors, cache entries, documents, graphs, analytics, catalog operations, migrations, transactions, CDC, and admin health.
• A native control plane for authentication, authorization, API keys, identity providers, tenants, notifications, analytics, storage, assets, WebRTC, and versioned policy distribution.
• A descriptor-driven contract: protos define services, tables, field security, endpoint security, SDK surfaces, event contracts, and generated docs.
• Multi-tenant request context on every call: tenant, project, user, purpose, service identity, scopes, correlation id, and client catalog version.
• SDKs for Go, Python, TypeScript/Node, Java, C#, and PHP/Laravel.
• A single CLI named udb for proto export, formatting, SDK generation, native service setup, app integration, local dev, broker serving, and diagnostics.

Current Surface

The service surface below is generated from the embedded descriptor — do not edit the counts by hand. Run udb native docs (which also regenerates docs/generated/native-services.md) and commit the refreshed block; the readme_services_block_matches_embedded_descriptor staleness test fails if it drifts.

Per-service RPC counts (native control plane):

0.3.7 Release Focus

UDB 0.3.7 is the current release. It carries the workflow-oriented simple-client SDK layer over the full 344-RPC surface (77 DataBroker RPCs plus 267 native control-plane RPCs) and folds in the post-v0.3.2 native-service wave, so normal application code stays short without hiding correctness rules — read-after-write, idempotency, tenant binding, and typed errors stay explicit or broker-owned.

• Simple-client facade across all SDKs: one connect + loginAndAdoptTenant, then storage.uploadFile / downloadFileBytes, data.table(...).select(...), authz.allowRole(...), metadata.afterWrite(receipt), and replay-safe typed retries with automatic idempotency keys.
• StorageService.DownloadFile server-streaming RPC with a presigned-default + streaming-fallback client helper, so file bytes never transit the broker on the common path.
• The full native-service wave is in source and service wiring: Vault, Lock, Scheduler, Webhook, Search, Cache, LiveQuery, Config, Metering, Backup, Embedding, and Workflow join the earlier Authn/Authz/IdP/Tenant/Notification/ Analytics/Storage/Asset/WebRTC/Control surfaces.
• Notification and analytics flows run through the native entity store path instead of hand-built SQL call sites; storage, asset, WebRTC, tenant, auth, IdP, and control services share the same native runtime/store binding and generated contract checks.
• SDKs are regenerated for Go, Python, TypeScript, Java, C#, and PHP, with cross-language conformance plus deep live coverage for the broker-backed SDK harnesses.
• CI release gates cover version consistency, native contract drift, SDK service coverage, MinIO-backed live SDK startup, the six-language scaffold-compiles job, and native-service live integration assumptions.

Since 0.3.7: hardening

The private masterplan/todo board re-grounded every tracked item in real v0.3.7 source and adversarially verified it against code anchors. The hardening wave (82 of the tracked items landed) includes:

• Verification depth — live backend-by-backend conformance for all nine canonical stores, with two real store bugs fixed (a promoted-primary read fence in Postgres wait_for_token, and SQL Server migration-audit backfill deferral).
• IR mediation by default — raw dispatch is gated, compiler classification is single-sourced, SDK IR builders ship in templates and committed SDKs, and served GenericDispatch cross-language byte parity plus the PG planner/IR merge oracle were observed green.
• Distributed correctness — Keeper-backed ClickHouse canonical CAS and Elasticsearch native CAS observed green; Qdrant fail-closed proof green; Weaviate/Pinecone terminally fail-closed (no usable CAS primitive); MySQL XA crash-recovery XA RECOVER fixed to the text protocol.
• Identity and compliance — SAML HTTP, internal-only listener gating, evidence export, and WebAuthn attestation-statement crypto (fixing a base64 ErrorDetail decode bug); enterprise token/key lifecycle, IdP/SAML/SCIM, and policy governance.
• Media plane — vendored ffmpeg transcode and the LiveKit SFU served WebRTC smoke observed green over the broker's three-listener topology.
• Native services — Metering QueryUsage RLS-GUC under-report fixed, and the Embedding backfill worker fixed (project-isolation filter plus two CDC journal-envelope read bugs) and proven live.
• Typed error and idempotency contract — a public udb.entity.v1.ErrorDetail wire trailer decoded across all six SDKs, and durable broker-side idempotency dedup with was_duplicate replay for keyed Upsert/Delete and BatchUpsert.

How It Feels

Your project owns the model:

syntax = "proto3";
package acme.billing.v1;

import "udb/core/common/v1/db.proto";

message Invoice {
  option (udb.core.common.v1.pg_table) = {
    schema: "billing"
    table: "invoices"
  };

  string invoice_id = 1 [(udb.core.common.v1.pg_column) = {
    primary_key: true
    sql_type: "text"
  }];

  string tenant_id = 2;
  string customer_id = 3;
  int64 total_cents = 4;
}

Application code calls the broker through an SDK:

from udb_client import Metadata, UdbClient, decode_records

meta = Metadata(
    tenant_id="acme",
    user_id="user-1",
    purpose="billing.api",
    scopes=("udb:read", "udb:write"),
    service_identity="billing-service",
    project_id="billing",
)

with UdbClient("127.0.0.1:50051", meta) as udb:
    rows = udb.select(message_type="acme.billing.v1.Invoice", limit=50)
    print(decode_records(rows))

Quick Start

Install the CLI from a release binary, through an SDK launcher, or from source:

cargo install --path .

Export UDB's shared protos into an application project:

udb proto export --fmt

Write app-owned protos that import UDB annotations:

import "udb/core/common/v1/db.proto";

Inspect the catalog and generated SQL:

udb catalog proto
udb sql proto
udb lint proto --human

Start a local broker:

udb serve proto "" 0.0.0.0:50051

Check runtime readiness:

udb doctor --human
udb compat-matrix

CLI

All public commands use the short binary name udb.

Backends

UDB knows these backend kinds:

postgres, mysql, sqlite, sqlserver, clickhouse, redis, memcached, qdrant, weaviate, pinecone, minio, s3, azureblob, gcs, mongodb, elasticsearch, neo4j, and cassandra.

The live capability matrix distinguishes compiled support from configured runtime availability:

udb compat-matrix

Native Control Plane

UDB ships a separate internal control-plane listener for native services. Bind it to an internal interface or place it behind a trusted gateway.

Details, including the read-your-writes consistency contract (write receipts, read fences, consistency modes) and idempotency was_duplicate replay: docs/native-services.md.

SDKs

Start here: sdk/README.md.

Examples

Documentation

• docs/README.md - documentation index
• docs/architecture.md - architecture, request flow, routing, pooling, events, and backend capability
• docs/api-rules.md - API route rules, OpenAPI operation IDs, and SDK alias policy
• docs/annotations.md - proto annotations
• docs/integration.md - application integration
• docs/native-services.md - native auth, authz, IdP, storage, assets, WebRTC, and SDK facades
• docs/operations.md - production readiness, config, runbooks, SLOs, and validation
• docs/security.md - request context, identity, authorization, sensitive data, and compliance profiles
• docs/testing.md - test commands and live checks
• VERSIONING.md - release and protocol versioning
• TESTING.md - top-level test guide

Development

Fast local checks:

cargo test --lib
buf lint
buf build
node scripts/check-versions.mjs

SDK conformance:

node sdk-conformance/run.mjs

The default Rust suite is designed to run without external services. Live backend, HA, and load checks are documented separately and require the matching infrastructure.

License

MIT. See LICENSE.