udb 0.2.0 - Docs.rs

UDB is a Rust implementation of a proto-driven data broker. It reads project-owned .proto schemas, extracts storage annotations, builds a catalog manifest, generates migration/bootstrap artifacts, and serves those schemas through a neutral gRPC DataBroker API — fronted by a native auth/authz control plane.

flowchart LR
    P["📦 project .proto<br/>(+ udb annotations)"] --> PA["🧩 parser →<br/>ProtoSchema AST"]
    PA --> CM["📚 CatalogManifest<br/>+ checksum"]
    CM --> GEN["🛠️ lint · drift ·<br/>migrations · SQL"]
    CM --> RT["⚙️ DataBroker runtime"]
    subgraph RT_PIPE["request pipeline"]
        direction LR
        AUTH["🔐 authn / authz"] --> ADM["🚦 channel admission"] --> IR["🔁 neutral IR"] --> EX["🔌 backend executor"]
    end
    RT --> RT_PIPE
    EX --> DB[("🗄️ 18 backends<br/>SQL · vector · object ·<br/>cache · doc · graph · column")]
    classDef accent fill:#1f6feb,stroke:#0b3d91,color:#fff;
    class RT,RT_PIPE accent;

This repo is not only a parser and not only a gRPC server. It is a crate, binary, runtime, protocol module, SDK workspace, backend plugin inventory, operation IR, migration engine, and a set of operational runbooks. It also has visible architectural history: early UDB was Postgres anchored, and the current codebase is in a peer-to-peer transition where canonical stores are explicit traits instead of implicit PgPool access.

What This Project Is

UDB tries to solve a specific problem: many services want to read and write business data, vectors, blobs, cache entries, CDC events, and admin/catalog state, but every service talking directly to every database creates drift in authorization, migrations, tenant isolation, observability, and retry behavior.

UDB centralizes those concerns:

Project schemas stay in normal project-owned proto packages.
UDB annotations describe relational tables, object fields, vector stores, caches, document stores, graph stores, time-series/column stores, and security.
The broker exposes one UDB-owned gRPC contract under proto/udb/....
Runtime requests carry tenant, purpose, scopes, service identity, project id, and catalog version metadata.
Backends are reached through a neutral logical IR and feature-gated plugin modules instead of service code hand-writing each database dialect.

⚡ Supported Features

Data plane (DataBroker, 73 RPCs):

Relational CRUD + batch (Select/BatchSelect/Upsert/BatchUpsert/Delete).
Vector search / hybrid search / upsert (Qdrant, Weaviate, Pinecone, Elasticsearch knn).
Object/blob put/get, presigned URLs, multipart (S3, MinIO, Azure Blob, GCS).
Cache get/set/delete/scan (Redis, Memcached).
Document / graph / time-series / analytical ops (MongoDB, Neo4j, ClickHouse, Cassandra).
Transactions: per-request transactionality, real Postgres 2PC and MySQL XA (UDB_2PC_ENABLED), sagas with recovery/compensation.
CDC → Kafka via a transactional outbox relay, with DLQ, topic policy, and a CDC control plane.
Catalog & migrations: staged/activate/rollback catalogs, proto-driven migration plan/apply with an audited op ledger.
Projections / materialized views, per-tenant RLS, field-level encryption (AES-256-GCM-SIV), rate limiting / fair channels / backpressure, Prometheus metrics.

Control plane (proto/udb/core/**, isolated listener — see Native Control Plane):

Authn: native JWT validation (JWKS/kid), UDB-issued RS256 JWT signing + refresh tokens, Argon2id passwords, RFC 6238 TOTP MFA, server-side sessions, CSRF, OTP, full user admin, mTLS + hybrid external identity (OIDC/Better Auth bridge).
Authz: RBAC + ABAC + simple ReBAC over a Casbin enforcer, role/policy/relationship CRUD, audit decisions, GetNativeAccess (restricted role + scoped DSN + RLS session vars), signed policy bundles for offline SDK caches.
ApiKey: hashed keys, scopes, rotation, revocation, usage stats.
Tenant / Notification / Analytics: tenant + config management, notification logs/templates/preferences/delivery-stats (with Kafka emit), and pipeline/executor/reconciliation/throughput/SLA analytics.

All native control-plane CRUD is proto-driven (table + column shape resolved from the embedded proto/udb/core/** manifest via NativeModel) and Postgres-backed, fail-closed — no in-memory stores.

Honest Status

The maintained docs now live under docs/README.md. The old older notes and duplicate runbooks were consolidated so current status is easier to verify. The source of truth for backend inventory is:

src/backend/mod.rs: BackendKind, tier, role, capability matrix, operation support.
src/backend/plugins/mod.rs: compiled plugin inventory.
src/runtime/executors/: runtime executor modules.
src/ir/compile/: backend-specific IR compilers.
Cargo.toml: default and optional feature graph.

The current code recognizes 18 backend kinds and the default feature set enables their plugin modules. A slim build can compile only the core pieces, for example --no-default-features --features postgres.

Also note that the Docker files still show some historical path assumptions in places. The Rust crate and CLI are the most reliable entry points while the repo split/packaging work settles.

Codebase Map

Approximate source shape at the time this README was written:

Area	Files	Purpose
`src/runtime`	102	Broker orchestration, service handlers, backend clients, CDC, catalog, system stores, security, metrics
`src/ir`	28	Neutral logical operations and backend compilers
`src/generation`	17	Manifest, SQL, DSN, drift, lint, and backend artifact generation
`src/migration`	7	Diffing, plans, audited apply, phase runner, db_ops sync
`src/control`	11	Startup lifecycle, FSM, hooks, notifications, approval workflow
`src/parser`	10	Hand-written proto lexer/parser and annotation extraction
`src/backend`	21	Backend identity, capabilities, plugin contract, plugin inventory
`src/cli`	7	`udb-proto-parser` command implementation
`src/planning`	4	Request planning helpers for broker operations
`src/schema`	3	Proto AST structs and deterministic checksums
`crates/udb-portable`	2	WASM/edge-safe parser/checksum/schema-cache subset

The public crate surface is collected in src/lib.rs. The binary entry point is tiny by design: src/main.rs calls the CLI module.

Request Flow

sequenceDiagram
    autonumber
    participant C as 📱 Client / SDK
    participant S as ⚙️ DataBrokerService
    participant A as 🔐 authz (v2)
    participant Ch as 🚦 channels
    participant IR as 🔁 IR + router
    participant B as 🔌 Backend
    C->>S: gRPC + metadata (tenant, purpose, scopes, …)
    S->>S: SecurityContext · ensure_ready() · catalog compat
    S->>A: authorize(identity, tenant, op, resource)
    A-->>S: Decision (allow + decision_id / deny)
    S->>Ch: acquire permit (limit · fairness · backpressure)
    Ch-->>S: permit
    S->>IR: lower to neutral IR · resolve target
    IR->>B: execute (SET LOCAL app.current_* for RLS)
    B-->>S: result
    S-->>C: response + catalog/consistency headers (+ write receipt)
    Note over S,B: side effects → metrics · audit · CDC · projection · saga · DLQ

For a normal gRPC call:

DataBrokerService receives the RPC in src/runtime/service/mod.rs.
The handler extracts metadata into a SecurityContext and request context.
ensure_ready() checks the startup lifecycle FSM has reached Completed.
Catalog compatibility is checked against x-udb-client-catalog-version.
ABAC policies evaluate service identity, tenant, purpose, operation, scopes, and message type.
A channel permit is acquired through src/runtime/channels.rs; this is where per-operation limits, fairness, and backpressure live.
The request is planned or lowered to neutral IR.
A backend target is resolved from project routing, target backend/instance, circuit breaker state, and plugin registry.
The backend executor runs the operation.
Responses include catalog/consistency headers; mutations also include a write receipt when possible.
Metrics, audit, CDC, projection, saga, or DLQ paths record side effects as configured.

The DataBroker data-plane contract defines 73 RPCs in proto/udb/services/v1/data_broker.proto. They cover relational, vector, object, cache, document, graph, time-series, analytical, transaction/2PC, CDC, resource admin, catalog, migration, DLQ, saga, policy, project, health, and admin/audit surfaces.

Alongside the data plane, UDB now ships a native control plane under proto/udb/core/** — six services (Authn, Authz, ApiKey, Tenant, Notification, Analytics, 77 RPCs total) that run on a separate, network-isolated listener (UDB_AUTH_GRPC_ADDR). See Native Control Plane.

Main Concepts

Project Protos

Project/application protos are schema input. They do not need to import or define the UDB DataBroker service. UDB parses annotations by suffix, so an annotation may be canonical like (udb.table) or project-qualified like (acme.billing.v1.table).

The parser supports:

table and column projections
primary keys, indexes, foreign keys, checks
RLS and tenant columns
vector/cache/object/document/graph/time-series/column/model-registry stores
proto3 reserved field names and ranges for drift safety
language options propagated into the manifest
annotation modes: compat, warn, strict

Key files:

Catalog Manifest

The catalog manifest is the broker's normalized view of parsed schemas. It is where proto messages become tables, columns, stores, projections, security metadata, language class names, checksums, warnings, and validation errors.

Key files:

Neutral IR

Data-plane operations lower into backend-neutral structs before compiler modules turn them into SQL, JSON HTTP payloads, key/value operations, object operations, or CQL/Cypher/etc.

The main IR operations are:

LogicalRead
LogicalWrite
LogicalDelete
LogicalSearch
LogicalAggregate
LogicalResourceOp

Key files:

🗄️ Backend Matrix

UDB separates backend identity from runtime availability:

BackendKind is the known backend enum.
BackendTier groups SQL/cache/vector/object/document/graph/column stores.
BackendRole says whether a backend can be canonical, projection-only, or both.
BackendCapability declares operation and consistency properties.
Backend plugin structs register backend-specific setup, generation, and conformance contracts.

The code declares 18 BackendKind variants (src/backend/mod.rs), all enabled in the default feature set. role is whether a backend can host UDB system tables (canonical) or only serve reads/projections; RLS is how per-tenant context is enforced (Postgres/MySQL/SQLite session GUCs, key-prefix for KV/object, filter predicate for document/vector). Slim builds compile a subset, e.g. --no-default-features --features postgres.

Backend	Tier	Feature flag	Role	Operations	Txn / 2PC	RLS
Postgres	SQL	`postgres` (always on)	canonical	relational CRUD, tx	yes / 2PC	session GUC
MySQL	SQL	`mysql`	canonical	relational CRUD, tx	yes / XA+2PC	session GUC
SQLite	SQL	`sqlite`	canonical	relational CRUD, tx	yes / —	context table
SQL Server	SQL	`mssql`	projection	relational CRUD, tx	yes / —	`SESSION_CONTEXT`
ClickHouse	column	`clickhouse`	both	analytical query, mutate	—	session setting
Redis	cache	`redis`	projection	cache get/set/del/scan	—	key prefix
Memcached	cache	`memcached`	projection	cache get/set	—	key prefix
Qdrant	vector	`qdrant`	projection	vector search/upsert	—	filter
Weaviate	vector	`weaviate`	projection	vector + hybrid search	—	filter
Pinecone	vector	`pinecone`	projection	vector + hybrid search	—	filter
MinIO	object	`s3`	projection	object put/get/presign	—	key prefix
S3	object	`s3`	projection	object put/get/presign	—	key prefix
Azure Blob	object	`azureblob`	projection	object put/get	—	key prefix
Google Cloud Storage	object	`gcs`	projection	object put/get	—	key prefix
MongoDB	document	`mongodb`	canonical	document find/upsert, tx	yes / —	filter
Elasticsearch	search	`elasticsearch`	projection	search + hybrid	—	filter
Neo4j	graph	`neo4j`	both	graph query/mutate, tx	yes / —	Cypher param
Cassandra / ScyllaDB	column	`cassandra`	projection	wide-column query/mutate	LWT only	partition key

Postgres is always compiled (never feature-gated); the other 17 are gated. MinIO and S3 share the s3 feature. src/backend/mod.rs is the source of truth for the full BackendCapability matrix (transactions, XA/2PC, RLS, vector/hybrid search, TTL, object-store, migration-ledger, consistency model).

Canonical Stores

This is the most important architectural transition in the repo.

Older UDB paths assumed Postgres was the canonical store for system tables, CDC, saga state, projection task state, migration audit, and consistency fences. The newer peer-to-peer work introduces:

CanonicalStore
DurabilityToken
SystemStores
CanonicalStoreRegistry
Postgres, MySQL, and SQLite implementations for system-store traits

Key files:

Do not read "universal DB layer" as "every backend has identical semantics." The code tries to be explicit about what compiles, what is unsupported, and what is eventually consistent or projection-only.

Runtime System Tables

UDB owns internal catalog/system tables for:

catalog versions and activation logs
project catalog bindings
migration runs and operation ledgers
CDC event journal, offsets, lock log, control table, topic policy, DLQ
saga coordinator
projection tasks
ABAC policies
admin audit log

Preview the DDL:

cargo run --bin udb-proto-parser -- system-ddl

Related files:

Repository Layout

Path	What lives there
`src/lib.rs`	Public library surface and compatibility re-exports
`src/main.rs`	Binary entry point
`src/cli`	CLI parsing and command handlers
`src/parser`	Proto lexer/parser and annotation extraction
`src/schema`	AST and checksum types
`src/generation`	Manifest/SQL/DSN/drift/lint generation
`src/ir`	Backend-neutral operation model and compilers
`src/backend`	Backend inventory, plugin trait, capability matrix
`src/runtime`	Broker runtime, service handlers, backend executors, CDC, security, metrics
`src/migration`	Migration diff/apply/sync/phase-runner
`src/control`	Startup lifecycle, FSM, approval, hooks, notifications
`proto`	UDB-owned gRPC/protobuf contract
`sdk`	Generated/wrapped clients
`examples`	Arbitrary project, multi-project, and toy plugin examples
`configs`	YAML config examples
`docs`	Operational docs, security, upgrade history, runbooks
`crates/udb-portable`	WASM/edge parser/checksum/schema-cache subset

Quick Start For Developers

The fastest meaningful flow is to use the arbitrary project example, because the UDB-owned protocol protos are service definitions, not domain schemas.

cargo test --lib

cargo run --bin udb-proto-parser -- lint examples/go_arbitary_project/proto --human
cargo run --bin udb-proto-parser -- catalog examples/go_arbitary_project/proto
cargo run --bin udb-proto-parser -- sql examples/go_arbitary_project/proto
cargo run --bin udb-proto-parser -- plan examples/go_arbitary_project/proto

Run a Postgres-backed broker locally:

Copy-Item .env.example .env.local
$env:UDB_PG_DSN = "postgresql://udb:udb@localhost:5432/udb?sslmode=prefer"
$env:UDB_ABAC_DEFAULT_ALLOW = "true"
cargo run --bin udb-proto-parser -- serve examples/go_arbitary_project/proto "" 0.0.0.0:50051

Run local readiness checks:

cargo run --bin udb-proto-parser -- doctor --human
cargo run --bin udb-proto-parser -- doctor --probe --human

CLI

The binary is udb-proto-parser. Its name is older than its current scope; it now drives parsing, generation, runtime serving, migration/admin checks, and the local playground.

Schema and planning:

cargo run --bin udb-proto-parser -- catalog <proto-root> [namespace]
cargo run --bin udb-proto-parser -- dsn <proto-root>
cargo run --bin udb-proto-parser -- sql <proto-root>
cargo run --bin udb-proto-parser -- plan <proto-root>
cargo run --bin udb-proto-parser -- lint <proto-root> --human
cargo run --bin udb-proto-parser -- drift <proto-root> --prior old_manifest.json
cargo run --bin udb-proto-parser -- explain <proto-root>
cargo run --bin udb-proto-parser -- manifest-export <proto-root>
cargo run --bin udb-proto-parser -- field-mask-preview <proto-root>

Runtime/admin:

cargo run --bin udb-proto-parser -- serve <proto-root> "" 0.0.0.0:50051
cargo run --bin udb-proto-parser -- doctor --probe --human
cargo run --bin udb-proto-parser -- health-check
cargo run --bin udb-proto-parser -- system-ddl
cargo run --bin udb-proto-parser -- tracker-ddl
cargo run --bin udb-proto-parser -- admin dry-run <proto-root>
cargo run --bin udb-proto-parser -- admin force-sync <proto-root>
cargo run --bin udb-proto-parser -- admin verify-audit --limit 250
cargo run --bin udb-proto-parser -- admin release-lock

Policy and compatibility:

$env:UDB_ABAC_POLICY_FILE = "docs/abac_seed.json"
cargo run --bin udb-proto-parser -- policy-lint
cargo run --bin udb-proto-parser -- policy-seed
cargo run --bin udb-proto-parser -- compat-matrix
cargo run --bin udb-proto-parser -- config-skeleton

Playground wrapper:

cargo run --bin udb-proto-parser -- dev up
cargo run --bin udb-proto-parser -- dev status
cargo run --bin udb-proto-parser -- dev logs udb
cargo run --bin udb-proto-parser -- dev smoke
cargo run --bin udb-proto-parser -- dev down

Configuration

Configuration is loaded as defaults plus optional file plus environment overlay. The standard config path is UDB_CONFIG_PATH; the complete operator template is .env.example. Env files are loaded in this order:

OS environment
.env.<APP_ENV>
.env.local
.env.prod
.env

Minimum required env for a normal Postgres-backed broker:

Variable	Meaning
`APP_ENV`	Selects `.env.<APP_ENV>` and labels the runtime environment
`UDB_ENV`	Security-mode switch; `production`/`prod` enables stricter defaults
`UDB_APP_NAME`	Broker/application identity
`UDB_PG_INSTANCES`	Named Postgres instances, usually `primary`
`UDB_PG_DSN_PRIMARY`	DSN for the named `primary` instance
`UDB_PG_DSN` or `DATABASE_URL`	Canonical primary Postgres DSN
`UDB_2PC_ENABLED`	Enables real Postgres prepared-transaction 2PC when `true`

Common optional env variables:

Variable	Meaning
`UDB_CONFIG_PATH`	YAML/JSON/TOML runtime config path
`UDB_BACKEND_INSTANCES`	Named backend instance descriptor list
`UDB_REDIS_DSN`	Redis cache/rate-limit/idempotency
`UDB_QDRANT_URL`	Qdrant vector backend
`UDB_MINIO_ENDPOINT`, `UDB_MINIO_ACCESS_KEY`, `UDB_MINIO_SECRET_KEY`	MinIO/S3-compatible object storage
`UDB_NOSQL_DSN`, `UDB_NOSQL_API_URL`	MongoDB/Atlas Data API backend
`UDB_GRAPH_DSN`, `UDB_GRAPH_HTTP_URL`	Neo4j graph backend
`UDB_COLUMN_DSN`, `UDB_COLUMN_HTTP_URL`	ClickHouse column backend
`UDB_KAFKA_BROKERS`	Kafka brokers for CDC
`UDB_ABAC_DEFAULT_ALLOW`	Development-only relaxed authorization
`UDB_ALLOW_DEGRADED_BACKENDS`	Allow startup with optional backend failures
`UDB_METRICS_ADDR`	Prometheus scrape address, default `0.0.0.0:50052`
`UDB_GRPC_ADDR`	Default serve address when not supplied positionally
`UDB_TLS_`, `UDB_MTLS_`	Server TLS and client CA config

See:

Security Model

UDB authorization is request-context based. Every non-health request should carry:

x-tenant-id
x-user-id
x-purpose
x-correlation-id
x-scopes
x-service-identity
x-udb-project-id
x-udb-client-catalog-version

The runtime supports:

JWT service identity
mTLS service identity
dev-only header fallback
ABAC policy evaluation
PII masking
field-level encryption
tenant-aware request context injection
audit logging
admin audit hash-chain verification
topic-policy enforcement for CDC

Start here:

Native Control Plane

Beyond the data-plane DataBroker, UDB serves a UDB-owned auth/admin control plane defined under proto/udb/core/**. These six services are network-isolated on a separate listener (UDB_AUTH_GRPC_ADDR, default loopback port+10) because they are a policy decision point that accepts the subject principal as input — they must not sit on the public DataBroker port where any client could assert identity. All of them are proto-driven (NativeModel) and Postgres-backed, failing closed when no PG pool is configured; their tables are generated from the embedded proto/udb/core/** manifest through the normal migration path.

flowchart TB
    APP["📱 app / SDK<br/>(6 languages)"]
    PEP["🛡️ trusted PEP / gateway"]
    subgraph UDB["UDB process"]
        direction TB
        PUB["🌐 public listener<br/><b>DataBroker</b> · 73 RPCs"]
        INT["🔒 internal listener · UDB_AUTH_GRPC_ADDR<br/><b>Authn · Authz · ApiKey · Tenant · Notification · Analytics</b>"]
    end
    BK[("🗄️ 18 backends")]
    KAFKA["📨 Kafka (CDC / events)"]
    APP -->|"data RPCs"| PUB
    PEP -->|"auth / admin RPCs"| INT
    INT -.->|"Authorize / GetNativeAccess"| PUB
    PUB --> BK
    INT --> BK
    PUB -.->|"outbox relay"| KAFKA
    INT -.->|"auth/notification events"| KAFKA
    classDef pub fill:#0e7490,stroke:#083344,color:#fff;
    classDef int fill:#7c3aed,stroke:#3b0764,color:#fff;
    class PUB pub;
    class INT int;

Service	Proto	RPCs	What it does
`AuthnService`	`core/authn`	23	Authenticate (JWT / session / API key / external), login/logout, RS256 JWT signing + refresh, sessions, TOTP MFA, CSRF, OTP, user admin
`AuthzService`	`core/authz`	23	`Authorize`/`CheckAccess`/batch over RBAC+ABAC+ReBAC (Casbin), role/policy/relationship CRUD, audit decisions, `GetNativeAccess`, `GetPolicyBundle`
`ApiKeyService`	`core/apikey`	7	Create/get/list/update/revoke/validate API keys + usage stats
`TenantService`	`core/tenant`	6	Tenant + tenant-config CRUD
`NotificationService`	`core/notification`	11	Notifications, templates, preferences, delivery stats (emits `udb.notification.sent.v1` to Kafka)
`AnalyticsService`	`core/analytics`	7	Pipeline metrics, executor performance, reconciliation, throughput, SLA compliance

Key capabilities:

Identity: native JWT (static PEM or JWKS URL with kid rotation), UDB-issued RS256 access tokens + refresh tokens (UDB_JWT_PRIVATE_KEY), Argon2id passwords (legacy keyed-HMAC auto-upgraded on login), RFC 6238 TOTP MFA, server-side sessions with idle/absolute TTL + revocation, mTLS SAN identity, and a hybrid external-identity bridge (the external provider proves who; UDB authz still decides what).
Authorization: one engine for RBAC (roles + bindings), ABAC (attribute conditions), and simple ReBAC (relationship tuples) with tenant/project domains, explicit-deny-wins, priority, and deterministic decision_id + audit records. UDB_AUTHZ_V2 (default on) routes broker enforcement through it.
Native fast path: GetNativeAccess authorizes a request and, when allowed, mints a short-lived restricted-role DSN plus the exact app.current_* session variables to SET LOCAL, so an SDK can talk to Postgres directly while the broker-generated RLS still applies.
Offline SDK authz: GetPolicyBundle returns an HMAC-signed, time-boxed snapshot the SDK caches to answer can() locally.

Source: src/runtime/authn/, src/runtime/authz/, src/runtime/service/auth_service/, docs/native-services.md.

Protocol And SDKs

The UDB-owned broker contract is:

The build script compiles those with tonic-build and writes a generated protocol.rs include under Cargo's OUT_DIR.

Generate SDKs:

.\scripts\gen_sdk.ps1

./scripts/gen_sdk.sh

SDK folders:

SDK	Path
Go	`sdk/go`
Python	`sdk/python`
TypeScript	`sdk/typescript`
C#	`sdk/csharp`
Java	`sdk/java`
PHP / Laravel	`sdk/php`

Protocol version: sdk/UDB_PROTOCOL_VERSION.

🚀 Quickstart Per Language

Most SDKs ship generated stubs (in each SDK's gen/ dir — no regen needed to consume), a thin broker client that attaches the request metadata headers (x-tenant-id, x-user-id, x-purpose, x-correlation-id, x-scopes, x-service-identity, x-udb-project-id, x-udb-client-catalog-version), and an auth client (Authenticate + Authorize/can). To regenerate after editing protos: buf generate (or scripts/gen_sdk.{ps1,sh}).

TypeScript note: the Node SDK (@udb_plus/sdk) loads the protos dynamically at runtime via @grpc/proto-loader (the .proto files are bundled into the package and resolved by protoRoot.ts) — you consume it through the package entry points (@udb_plus/sdk, /client, /auth), not by importing the gen/ stubs. The committed sdk/typescript/gen/** tree is a buf drift-parity artifact (kept in lockstep with the protos by CI) and is intentionally excluded from the published package and the build; it requires @bufbuild/protobuf and is not part of the SDK's runtime. See sdk/typescript/gen/README.md.

import (
    entityv1 "github.com/fahara02/udb/sdk/go/gen/udb/entity/v1"
    authzv1 "github.com/fahara02/udb/sdk/go/gen/udb/core/authz/services/v1"
    "github.com/fahara02/udb/sdk/go/udbclient"
    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials/insecure"
)

conn, _ := grpc.NewClient("localhost:50051", grpc.WithTransportCredentials(insecure.NewCredentials()))
meta := udbclient.Metadata{TenantID: "acme", UserID: "user-1", Purpose: "web.request",
    Scopes: []string{"udb:read", "udb:write"}, ServiceIdentity: "billing.api",
    ClientCatalogVersion: udbclient.ProtocolVersion}

udb := udbclient.New(conn, meta)
rs, _ := udb.Select(ctx, &entityv1.SelectRequest{MessageType: "acme.billing.v1.Invoice", Limit: 50})

auth := udbclient.NewAuthClient(conn, meta)
allowed, decision, _ := auth.Can(ctx, &authzv1.ResourceRef{MessageType: "acme.billing.v1.Invoice"}, "read", "")
// native fast path: grant, _ := auth.NativeAccess(ctx, res, "data.select", ""); udbclient.WithNativeTx(ctx, db, grant, fn)

from udb_client import Metadata, UdbClient, decode_records
from udb_client.auth import UdbAuthClient
from udb.core.authz.services.v1 import core_pb2 as authz

meta = Metadata(tenant_id="acme", user_id="user-1", purpose="billing.demo",
                correlation_id="demo-001", scopes=("udb:read", "udb:write"))

with UdbClient("127.0.0.1:50051", meta) as udb:
    udb.upsert(message_type="acme.billing.v1.Customer",
               record={"customer_id": "cus_001", "tenant_id": "acme"},
               conflict_fields=("customer_id",))
    rs = udb.select(message_type="acme.billing.v1.Customer", limit=10)
    print(decode_records(rs))

with UdbAuthClient("127.0.0.1:50051", meta) as auth:
    allowed, decision = auth.can(authz.ResourceRef(message_type="acme.billing.v1.Customer"), "read")

import { dataBrokerClient, metadata, UdbMetadata } from "@udb_plus/sdk/client";
import { UdbAuthClient } from "@udb_plus/sdk/auth";

const meta: UdbMetadata = { tenantId: "acme", userId: "user-1", purpose: "web.request",
    scopes: ["udb:read", "udb:write"], serviceIdentity: "billing.api" };

const broker = dataBrokerClient("localhost:50051");
broker.Select({ message_type: "acme.billing.v1.Invoice", limit: 50 }, metadata(meta),
    (err: any, rs: any) => console.log(rs?.records));

const auth = new UdbAuthClient("localhost:50051", meta);
const [allowed, decision] = await auth.can({ message_type: "acme.billing.v1.Invoice" }, "read");

import dev.udb.client.*;
import com.udb.entity.v1.Types.*;
import com.udb.core.authz.services.v1.ResourceRef;

var meta = new UdbMetadata("acme", "web.request", "corr-123",
    java.util.List.of("udb:read", "udb:write"), "billing.api", "user-1", "default", UdbClient.PROTOCOL_VERSION);

try (UdbClient udb = new UdbClient("localhost:50051", meta)) {
    RecordSet rs = udb.select(SelectRequest.newBuilder()
        .setMessageType("acme.billing.v1.Invoice").setLimit(50).build());
}
try (UdbAuthClient auth = new UdbAuthClient("localhost:50051", meta)) {
    var d = auth.can(ResourceRef.newBuilder().setMessageType("acme.billing.v1.Invoice").build(), "read", "");
}

using Udb.Client; using Udb.Entity.V1;
using AuthzV1 = udb.core.Authz.Services.V1;

await using var udb = new UdbClient("http://localhost:50051", new UdbMetadata(
    TenantId: "acme", Purpose: "web.request", CorrelationId: "corr-123",
    Scopes: new[] { "udb:read", "udb:write" }, ServiceIdentity: "billing.api", UserId: "user-1"));
RecordSet rs = await udb.SelectAsync(new SelectRequest { MessageType = "acme.billing.v1.Invoice", Limit = 50 });

await using var auth = new UdbAuthClient("http://localhost:50051", /* same meta */ default!);
var (allowed, decision) = await auth.CanAsync(new AuthzV1.ResourceRef { MessageType = "acme.billing.v1.Invoice" }, "read");

use Fahara02\UdbLaravel\Facades\Udb;
use Udb\Entity\V1\SelectRequest;
use Udb\Core\Authz\Services\V1\ResourceRef;

// request context auto-bound by middleware; pass UdbMetadata explicitly off-request
$rs = Udb::select((new SelectRequest())->setMessageType('acme.billing.v1.Invoice')->setLimit(50));

[$allowed, $decision] = app(\Fahara02\UdbLaravel\UdbAuthClient::class)
    ->can((new ResourceRef())->setMessageType('acme.billing.v1.Invoice'), 'read');

Native fast-path transaction helpers (WithNativeTx/native_transaction/withNativeTx) and a local TTL authz cache (AuthzCache) ship in the Go, Python, and TypeScript SDKs; C#/Java/PHP expose nativeAccess + getPolicyBundle and apply the grant's set_config session vars manually. Full per-language detail: each SDK's README.

Testing

Fast local tests:

cargo test --lib

Backend feature sweeps:

cargo test --all-features --lib
cargo test --no-default-features --features postgres --lib
cargo test --features clickhouse,mssql,cassandra --lib

Proto contract:

buf lint
buf build
buf generate

Integration tests are opt-in:

docker compose -f docker-compose.integration.yml up -d --wait
$env:UDB_INTEGRATION_TESTS = "1"
cargo test --test integration_tests -- --nocapture
docker compose -f docker-compose.integration.yml down -v --remove-orphans

The full default Rust suite is meant to run without external services. Live Docker/infrastructure tests are guarded by env variables or #[ignore].

See:

Load, Soak, And Operations

Load profiles are scripted through ghz:

$env:UDB_HOST = "localhost:50051"
$env:CONCURRENCY = "50"
$env:TOTAL_REQUESTS = "10000"
$env:PROFILE = "read-heavy"
.\scripts\load_test.ps1

Profiles include:

read-heavy
write-heavy
mixed-projection
tenant-noisy-neighbor
backend-outage
reload-during-traffic
multi-project-smoke

Operational docs:

Topic	Document
Docs index	`docs/README.md`
Architecture and backend inventory	`docs/architecture.md`
Operations, topology, reload, backup, and load profiles	`docs/operations.md`
Security, audit, encryption, and supply chain	`docs/security.md`
Testing and live acceptance	`docs/testing.md`

Examples

Example	What to look at
`examples/go_arbitary_project`	A Go project namespace UDB does not own; shows table, cache, vector, object, PII, encryption end-to-end
`examples/python_arbitary_project`	The same arbitrary-project flow driven from the Python SDK
`examples/php_arbitary_project`	The same flow from the PHP/Laravel SDK
`examples/native-services/go`	Using the native control plane (Authn/Authz/ApiKey/Tenant/Notification/Analytics) from Go
`examples/multi_project`	One broker serving unrelated projects with separate proto roots/catalogs
`examples/toy_backend_plugin`	Minimal external backend plugin contract

Portable Crate

crates/udb-portable is the browser/edge-safe subset. It path-includes the same AST, checksum, lexer, and parser source files used by the main crate. It deliberately excludes tokio, sqlx, tonic, cloud SDKs, Kafka, Redis, and filesystem directory parsing.

Use it when a client or edge worker needs to parse proto source, compute the same schema checksum as the server, or track catalog/schema compatibility without embedding the whole broker.

Kubernetes

deploy/kubernetes contains CRD contracts for:

UdbBroker
UdbProjectCatalog
UdbBackendInstance
UdbMigrationRun
UdbCdcStream
UdbProjectionWorker

Apply contracts:

kubectl apply -f deploy/kubernetes/crds/udb.io_crds.yaml

These are controller-neutral contracts. The repo contains CRDs, not a complete operator implementation.

Supply Chain

The intended gate is:

cargo deny check advisories bans licenses sources

The policy denies unknown registries, git dependencies, and undocumented source exceptions. See docs/security.md.

Known Rough Edges

Some newer backend plugins are still plugin-owned rather than fully covered by one universal connection lifecycle.
Disabled-feature reporting should be aligned for every backend plugin.
Some Docker/package paths still reflect older monorepo layouts.
The default build intentionally pulls many backend SDKs; use slim feature builds to check dependency hygiene.
Several live acceptance gates in the docs require real infrastructure and are not satisfied by code-only tests.
The crate currently warns on unused/dead code during build; the warnings are tracked by the refactor history and are not treated as fatal yet.

Where To Start When Changing Code

Task	Start here
Add or change proto annotation parsing	`src/parser/options.rs`, `src/parser/db_parser.rs`, `src/schema/ast.rs`
Add a backend operation	`src/ir/operations.rs`, `src/ir/compile`, `src/runtime/executors`
Add a backend plugin	`src/backend/plugin.rs`, `src/backend/plugins`, `examples/toy_backend_plugin`
Change gRPC behavior	`proto/udb/services/v1/data_broker.proto`, `src/runtime/service`
Change auth or metadata	`src/runtime/security.rs`, `src/runtime/service/mod.rs`, `src/embedded.rs`
Change catalog/migration behavior	`src/generation/manifest`, `src/migration`, `src/control/lifecycle.rs`
Change system-store behavior	`src/runtime/canonical_store`, `src/runtime/system.rs`
Change config loading	`src/runtime/config`, `src/cli/env_setup.rs`, `build.rs`