<p align="center">
<img src="docs/assets/udb_logo.svg" alt="UDB logo" width="160">
</p>
<h1 align="center">Universal Data Broker</h1>
<p align="center">
<strong>UDB :: Universal Data Broker</strong><br>
<sub>gRPC data plane | native control plane | tenant/project scope guard<br>crate v0.4.0 | protocol v1.0.0</sub>
</p>
<p align="center">
<a href="Cargo.toml"><img alt="Rust 2024" src="https://img.shields.io/badge/Rust-2024-b7410e?logo=rust&logoColor=white"></a>
<a href="proto/README.md"><img alt="gRPC + Protobuf" src="https://img.shields.io/badge/API-gRPC%20%2B%20Protobuf-2563eb?logo=grpc&logoColor=white"></a>
<a href="sdk/UDB_PROTOCOL_VERSION"><img alt="Protocol 1.0.0" src="https://img.shields.io/badge/protocol-1.0.0-059669"></a>
<a href="#sdks"><img alt="SDKs" src="https://img.shields.io/badge/SDKs-Go%20%C2%B7%20Python%20%C2%B7%20TypeScript%20%C2%B7%20Java%20%C2%B7%20C%23%20%C2%B7%20PHP-334155"></a>
<a href="LICENSE"><img alt="License MIT" src="https://img.shields.io/badge/license-MIT-555"></a>
</p>
UDB is a public, schema-driven knowledge graph and data broker for applications
that need one typed API in front of many data systems.
You describe your domain in normal `.proto` files, add UDB annotations for
storage and security behavior, then run one broker in front of relational
databases, object stores, caches, vector stores, document stores, graph stores,
analytics systems, and native identity services. Application code calls UDB
through gRPC or an SDK; UDB handles routing, metadata, authorization, migrations,
CDC, and backend-specific execution.
<p align="center">
<img src="docs/assets/architecture-pipeline.svg" alt="UDB architecture pipeline from proto files to catalog generation, runtime authorization, routing, and backend execution" width="940">
</p>
## What UDB Gives You
- One `DataBroker` API for relational records, objects, vectors, cache entries,
documents, graphs, analytics, catalog operations, migrations, transactions,
CDC, and admin health.
- A native control plane for authentication, authorization, API keys, identity
providers, tenants, notifications, analytics, storage, assets, WebRTC, and
versioned policy distribution.
- A descriptor-driven contract: protos define services, tables, field security,
endpoint security, SDK surfaces, event contracts, and generated docs.
- Multi-tenant request context on every call: tenant, project, user, purpose,
service identity, scopes, correlation id, and client catalog version.
- SDKs for Go, Python, TypeScript/Node, Java, C#, and PHP/Laravel.
- A single CLI named `udb` for proto export, formatting, SDK generation, native
service setup, app integration, local dev, broker serving, and diagnostics.
## Current Surface
The service surface below is generated from the embedded descriptor — do not
edit the counts by hand. Run `udb native docs` (which also regenerates
[docs/generated/native-services.md](docs/generated/native-services.md)) and
commit the refreshed block; the `readme_services_block_matches_embedded_descriptor`
staleness test fails if it drifts.
| Data plane | 77 `DataBroker` RPCs |
| Native control plane | 27 services, 267 RPCs |
Per-service RPC counts (native control plane):
| `analytics` | 7 |
| `apikey` | 9 |
| `asset` | 8 |
| `authn` | 50 |
| `authz` | 41 |
| `backup` | 8 |
| `cache` | 7 |
| `config` | 5 |
| `control` | 6 |
| `embedding` | 6 |
| `idp` | 27 |
| `livequery` | 1 |
| `lock` | 3 |
| `metering` | 6 |
| `notification` | 12 |
| `scheduler` | 6 |
| `search` | 5 |
| `storage` | 8 |
| `tenant` | 7 |
| `vault` | 14 |
| `webhook` | 6 |
| `webrtc_peer` | 5 |
| `webrtc_room` | 9 |
| `webrtc_signaling` | 1 |
| `webrtc_track` | 4 |
| `webrtc_turn` | 1 |
| `workflow` | 5 |
| Contract manifest | 733 messages, 49 table-backed models, 192 event contracts |
| Backends | 18 backend kinds across SQL, cache, vector, object, document, graph, and column stores |
| SDKs | Go, Python, TypeScript/Node, Java, C#, PHP/Laravel |
| Release | crate/SDK version `0.4.0`, wire protocol `1.0.0` |
## 0.3.7 Release Focus
UDB 0.3.7 is the current release. It carries the workflow-oriented simple-client
SDK layer over the full **344-RPC surface** (77 `DataBroker` RPCs plus 267 native
control-plane RPCs) and folds in the post-v0.3.2 native-service wave, so normal
application code stays short without hiding correctness rules — read-after-write,
idempotency, tenant binding, and typed errors stay explicit or broker-owned.
- Simple-client facade across all SDKs: one `connect` + `loginAndAdoptTenant`,
then `storage.uploadFile` / `downloadFileBytes`, `data.table(...).select(...)`,
`authz.allowRole(...)`, `metadata.afterWrite(receipt)`, and replay-safe typed
retries with automatic idempotency keys.
- `StorageService.DownloadFile` server-streaming RPC with a presigned-default +
streaming-fallback client helper, so file bytes never transit the broker on the
common path.
- The full native-service wave is in source and service wiring: Vault, Lock,
Scheduler, Webhook, Search, Cache, LiveQuery, Config, Metering, Backup,
Embedding, and Workflow join the earlier Authn/Authz/IdP/Tenant/Notification/
Analytics/Storage/Asset/WebRTC/Control surfaces.
- Notification and analytics flows run through the native entity store path
instead of hand-built SQL call sites; storage, asset, WebRTC, tenant, auth, IdP,
and control services share the same native runtime/store binding and generated
contract checks.
- SDKs are regenerated for Go, Python, TypeScript, Java, C#, and PHP, with
cross-language conformance plus deep live coverage for the broker-backed SDK
harnesses.
- CI release gates cover version consistency, native contract drift, SDK service
coverage, MinIO-backed live SDK startup, the six-language `scaffold-compiles`
job, and native-service live integration assumptions.
### Since 0.3.7: hardening
The private masterplan/todo board re-grounded every tracked item in real v0.3.7
source and adversarially verified it against code anchors. The hardening wave (82 of the
tracked items landed) includes:
- **Verification depth** — live backend-by-backend conformance for all nine
canonical stores, with two real store bugs fixed (a promoted-primary read fence
in Postgres `wait_for_token`, and SQL Server migration-audit backfill deferral).
- **IR mediation by default** — raw dispatch is gated, compiler classification is
single-sourced, SDK IR builders ship in templates and committed SDKs, and served
`GenericDispatch` cross-language byte parity plus the PG planner/IR merge oracle
were observed green.
- **Distributed correctness** — Keeper-backed ClickHouse canonical CAS and
Elasticsearch native CAS observed green; Qdrant fail-closed proof green;
Weaviate/Pinecone terminally fail-closed (no usable CAS primitive); MySQL XA
crash-recovery `XA RECOVER` fixed to the text protocol.
- **Identity and compliance** — SAML HTTP, internal-only listener gating, evidence
export, and WebAuthn attestation-statement crypto (fixing a base64 ErrorDetail
decode bug); enterprise token/key lifecycle, IdP/SAML/SCIM, and policy
governance.
- **Media plane** — vendored ffmpeg transcode and the LiveKit SFU served WebRTC
smoke observed green over the broker's three-listener topology.
- **Native services** — Metering `QueryUsage` RLS-GUC under-report fixed, and the
Embedding backfill worker fixed (project-isolation filter plus two CDC
journal-envelope read bugs) and proven live.
- **Typed error and idempotency contract** — a public `udb.entity.v1.ErrorDetail`
wire trailer decoded across all six SDKs, and durable broker-side idempotency
dedup with `was_duplicate` replay for keyed Upsert/Delete and BatchUpsert.
## How It Feels
Your project owns the model:
```proto
syntax = "proto3";
package acme.billing.v1;
import "udb/core/common/v1/db.proto";
message Invoice {
option (udb.core.common.v1.pg_table) = {
schema: "billing"
table: "invoices"
};
string invoice_id = 1 [(udb.core.common.v1.pg_column) = {
primary_key: true
sql_type: "text"
}];
string tenant_id = 2;
string customer_id = 3;
int64 total_cents = 4;
}
```
Application code calls the broker through an SDK:
```python
from udb_client import Metadata, UdbClient, decode_records
meta = Metadata(
tenant_id="acme",
user_id="user-1",
purpose="billing.api",
scopes=("udb:read", "udb:write"),
service_identity="billing-service",
project_id="billing",
)
with UdbClient("127.0.0.1:50051", meta) as udb:
rows = udb.select(message_type="acme.billing.v1.Invoice", limit=50)
print(decode_records(rows))
```
## Quick Start
Install the CLI from a release binary, through an SDK launcher, or from source:
```bash
cargo install --path .
```
Export UDB's shared protos into an application project:
```bash
udb proto export --fmt
```
Write app-owned protos that import UDB annotations:
```proto
import "udb/core/common/v1/db.proto";
```
Inspect the catalog and generated SQL:
```bash
udb catalog proto
udb sql proto
udb lint proto --human
```
Start a local broker:
```bash
udb serve proto "" 0.0.0.0:50051
```
Check runtime readiness:
```bash
udb doctor --human
udb compat-matrix
```
## CLI
All public commands use the short binary name `udb`.
| `udb init` | Build or preview a project-aware UDB setup plan |
| `udb proto export --fmt` | Export UDB annotation and broker protos into an app |
| `udb proto fmt --check` | Check annotation formatting |
| `udb catalog`, `udb sql`, `udb plan`, `udb drift` | Inspect schemas, SQL, migrations, and drift |
| `udb sdk generate` | Generate descriptor-aware SDK surfaces from templates |
| `udb native list`, `udb native manifest`, `udb native docs` | Inspect native-service contracts |
| `udb app init` | Add a language/framework integration scaffold |
| `udb dev up`, `udb dev smoke` | Run and test the local playground |
| `udb serve` | Start the gRPC broker |
| `udb doctor`, `udb health-check` | Check runtime and dependency readiness |
| `udb auth ...` | Use control-plane helpers for principals, sessions, API keys, roles, relations, and policies |
| `udb dbops sync` | Sync generated database-operation artifacts |
## Backends
UDB knows these backend kinds:
`postgres`, `mysql`, `sqlite`, `sqlserver`, `clickhouse`, `redis`, `memcached`,
`qdrant`, `weaviate`, `pinecone`, `minio`, `s3`, `azureblob`, `gcs`, `mongodb`,
`elasticsearch`, `neo4j`, and `cassandra`.
The live capability matrix distinguishes compiled support from configured
runtime availability:
```bash
udb compat-matrix
```
## Native Control Plane
UDB ships a separate internal control-plane listener for native services. Bind
it to an internal interface or place it behind a trusted gateway.
| Authn | Login, sessions, refresh tokens, JWT/JWKS, MFA, OTP, devices, WebAuthn |
| Authz | RBAC, ABAC, ReBAC, decisions, policy bundles, native access, governance |
| API keys | Hashed keys, scopes, rotation, revocation, usage stats |
| Identity providers | OIDC, SAML, SCIM, JIT, external identity links |
| Control distribution | Versioned policy/resource distribution with ACK/NACK |
| Tenant, notification, analytics | Tenant config, messages, templates, metrics, SLA views |
| Storage and asset | Object metadata, presigned URLs, asset pipelines, vector-ready workflows |
| WebRTC | Rooms, peers, tracks, TURN credentials, signaling, egress/SFU |
| Vault | Secrets, transit encrypt/sign/HMAC, dynamic database credentials |
| Metering | Usage recording and per-tenant quotas |
| Scheduler, workflow, lock | Cron/one-shot jobs, durable sagas, distributed locks with fencing tokens |
| Search, embedding | Managed search indexes, vector retrieval, and backfill enumeration |
| Webhook | SSRF-guarded outbound webhook endpoints and delivery tracking |
| Config, live query | Feature flags and server-streaming live-query subscriptions |
| Backup | Tenant-scoped backup and restore |
Details, including the read-your-writes consistency contract (write receipts,
read fences, consistency modes) and idempotency `was_duplicate` replay:
[docs/native-services.md](docs/native-services.md).
## SDKs
| Go | `go get github.com/fahara02/udb/sdk/go@v0.4.0` |
| Python | `pip install udb-client==0.4.0` |
| TypeScript / Node | `npm i @udb_plus/sdk@0.4.0` |
| PHP / Laravel | `composer require fahara02/udb-laravel:^0.4.0` |
| C# | `dotnet add package Udb.Client --version 0.4.0` |
| Java | `dev.udb:udb-java-client` (`0.4.0` target; build from checkout until publishing lands) |
Start here: [sdk/README.md](sdk/README.md).
## Examples
| [examples/go_arbitary_project](examples/go_arbitary_project/README.md) | Go app with app-owned protos and UDB broker calls |
| [examples/python_arbitary_project](examples/python_arbitary_project/README.md) | Python app flow with generated models and SDK calls |
| [examples/php_arbitary_project](examples/php_arbitary_project/README.md) | PHP/Laravel app flow |
| [examples/php_quickstart](examples/php_quickstart/README.md) | CRUD, authz scopes, and relationships |
| [examples/native-services](examples/native-services/README.md) | Native auth/authz/API-key/native-access examples |
| [examples/multi_project](examples/multi_project/README.md) | One broker serving separate project catalogs |
| [examples/toy_backend_plugin](examples/toy_backend_plugin/README.md) | Minimal external backend plugin |
## Documentation
- [docs/README.md](docs/README.md) - documentation index
- [docs/architecture.md](docs/architecture.md) - architecture, request flow, routing, pooling, events, and backend capability
- [docs/api-rules.md](docs/api-rules.md) - API route rules, OpenAPI operation IDs, and SDK alias policy
- [docs/annotations.md](docs/annotations.md) - proto annotations
- [docs/integration.md](docs/integration.md) - application integration
- [docs/native-services.md](docs/native-services.md) - native auth, authz, IdP, storage, assets, WebRTC, and SDK facades
- [docs/operations.md](docs/operations.md) - production readiness, config, runbooks, SLOs, and validation
- [docs/security.md](docs/security.md) - request context, identity, authorization, sensitive data, and compliance profiles
- [docs/testing.md](docs/testing.md) - test commands and live checks
- [VERSIONING.md](VERSIONING.md) - release and protocol versioning
- [TESTING.md](TESTING.md) - top-level test guide
## Development
Fast local checks:
```bash
cargo test --lib
buf lint
buf build
node scripts/check-versions.mjs
```
SDK conformance:
```bash
node sdk-conformance/run.mjs
```
The default Rust suite is designed to run without external services. Live
backend, HA, and load checks are documented separately and require the matching
infrastructure.
## License
MIT. See [LICENSE](LICENSE).