<p align="center">
<img src="docs/assets/udb_logo.svg" alt="UDB logo" width="160">
</p>
<h1 align="center">Universal Data Broker</h1>
<p align="center">
<strong>UDB :: Universal Data Broker</strong><br>
<sub>gRPC data plane | native control plane | tenant/project scope guard<br>crate v0.4.13 | protocol v1.0.0</sub>
</p>
<p align="center">
<a href="Cargo.toml"><img alt="Rust 2024" src="https://img.shields.io/badge/Rust-2024-b7410e?logo=rust&logoColor=white"></a>
<a href="proto/README.md"><img alt="gRPC + Protobuf" src="https://img.shields.io/badge/API-gRPC%20%2B%20Protobuf-2563eb?logo=grpc&logoColor=white"></a>
<a href="sdk/UDB_PROTOCOL_VERSION"><img alt="Protocol 1.0.0" src="https://img.shields.io/badge/protocol-1.0.0-059669"></a>
<a href="#sdks"><img alt="SDKs" src="https://img.shields.io/badge/SDKs-Go%20%C2%B7%20Python%20%C2%B7%20TypeScript%20%C2%B7%20Java%20%C2%B7%20C%23%20%C2%B7%20PHP-334155"></a>
<a href="LICENSE"><img alt="License MIT" src="https://img.shields.io/badge/license-MIT-555"></a>
</p>
UDB is a public, schema-driven knowledge graph and data broker for applications
that need one typed API in front of many data systems.
You describe your domain in normal `.proto` files, add UDB annotations for
storage and security behavior, then run one broker in front of relational
databases, object stores, caches, vector stores, document stores, graph stores,
analytics systems, and native identity services. Application code calls UDB
through gRPC or an SDK; UDB handles routing, metadata, authorization, migrations,
CDC, and backend-specific execution.
<p align="center">
<img src="docs/assets/architecture-pipeline.svg" alt="UDB architecture pipeline from proto files to catalog generation, runtime authorization, routing, and backend execution" width="940">
</p>
## What UDB Gives You
- One `DataBroker` API for relational records, objects, vectors, cache entries,
documents, graphs, analytics, catalog operations, migrations, transactions,
CDC, and admin health.
- A native control plane for authentication, authorization, API keys, identity
providers, tenants, notifications, analytics, storage, assets, WebRTC, and
versioned policy distribution.
- A descriptor-driven contract: protos define services, tables, field security,
endpoint security, SDK surfaces, event contracts, and generated docs.
- Multi-tenant request context on every call: tenant, project, user, purpose,
service identity, scopes, correlation id, and client catalog version.
- SDKs for Go, Python, TypeScript/Node, Java, C#, and PHP/Laravel.
- A single CLI named `udb` for proto export, formatting, SDK generation, native
service setup, app integration, local dev, broker serving, and diagnostics.
## Current Surface
The service surface below is generated from the embedded descriptor — do not
edit the counts by hand. Run `udb native docs` (which also regenerates
[docs/generated/native-services.md](docs/generated/native-services.md)) and
commit the refreshed block; the `readme_services_block_matches_embedded_descriptor`
staleness test fails if it drifts.
| Data plane | 77 `DataBroker` RPCs |
| Native control plane | 27 services, 267 RPCs |
Per-service RPC counts (native control plane):
| `analytics` | 7 |
| `apikey` | 9 |
| `asset` | 8 |
| `authn` | 50 |
| `authz` | 41 |
| `backup` | 8 |
| `cache` | 7 |
| `config` | 5 |
| `control` | 6 |
| `embedding` | 6 |
| `idp` | 27 |
| `livequery` | 1 |
| `lock` | 3 |
| `metering` | 6 |
| `notification` | 12 |
| `scheduler` | 6 |
| `search` | 5 |
| `storage` | 8 |
| `tenant` | 7 |
| `vault` | 14 |
| `webhook` | 6 |
| `webrtc_peer` | 5 |
| `webrtc_room` | 9 |
| `webrtc_signaling` | 1 |
| `webrtc_track` | 4 |
| `webrtc_turn` | 1 |
| `workflow` | 5 |
| Contract manifest | 733 messages, 49 table-backed models, 192 event contracts |
| Backends | 18 backend kinds across SQL, cache, vector, object, document, graph, and column stores |
| SDKs | Go, Python, TypeScript/Node, Java, C#, PHP/Laravel |
| Release | crate/SDK version `0.4.13`, wire protocol `1.0.0` |
## What's Battle-Tested
UDB isn't a prototype. Every backend it advertises is exercised by live
conformance tests against a real server, and the hard parts of putting one API in
front of many databases are already handled for you:
- **Multi-tenant isolation that actually holds.** Row-level security is enforced
in the database, scoped by a canonical tenant **UUID** — not a string you can
typo. The SDKs resolve your human-readable tenant *code* (`"acme"`) to that UUID
for you, which removes the single most common first-integration bug: a code/UUID
mismatch that silently returns zero rows.
- **Correctness you don't re-implement.** Read-after-write, idempotent retries
(keyed `Upsert`/`Delete` replay safely), typed errors on a stable wire contract,
and tenant/project binding are broker-owned. Your application code stays short
without the rules going quiet.
- **A real control plane, not stubs.** Native identity (authentication,
authorization, API keys, IdP with SAML/SCIM), plus Vault, Lock, Scheduler,
Webhook, Search, Cache, LiveQuery, Config, Metering, Backup, Embedding,
Workflow, Storage, Assets, and WebRTC — all served through the same
tenant-scoped runtime.
- **Broad backend coverage, one typed API.** Relational (Postgres, MySQL, SQLite,
SQL Server), document (MongoDB), vector (Qdrant, Weaviate, Pinecone), object
(S3, MinIO, GCS, Azure Blob), cache (Redis, Memcached), graph (Neo4j),
wide-column (Cassandra), analytics (ClickHouse), and search (Elasticsearch).
- **Six SDKs from one contract.** Go, Python, TypeScript, Java, C#, and PHP are
generated from the same protos and checked for cross-language parity, so an
`Invoice` behaves the same in every language — and the full RPC surface is
measured live in each.
## How It Feels
Your project owns the model:
```proto
syntax = "proto3";
package acme.billing.v1;
import "udb/core/common/v1/db.proto";
message Invoice {
option (udb.core.common.v1.pg_table) = {
schema: "billing"
table: "invoices"
};
string invoice_id = 1 [(udb.core.common.v1.pg_column) = {
primary_key: true
sql_type: "text"
}];
string tenant_id = 2;
string customer_id = 3;
int64 total_cents = 4;
}
```
Application code calls the broker through an SDK:
```python
from udb_client import Metadata, UdbClient, decode_records
meta = Metadata(
tenant_id="acme", # your human tenant code; the SDK login flow resolves it to
# the canonical UUID that RLS checks — see examples/python_enterprise
# (passing a raw code is the #1 "why are my reads empty?" bug).
user_id="user-1",
purpose="billing.api",
scopes=("udb:read", "udb:write"),
service_identity="billing-service",
project_id="billing",
)
with UdbClient("127.0.0.1:50051", meta) as udb:
rows = udb.select(message_type="acme.billing.v1.Invoice", limit=50)
print(decode_records(rows))
```
## Quick Start
The whole idea: **you describe your data model in protobuf, and UDB turns it into
a running, multi-tenant, RLS-protected API over your database — no ORM, no
hand-written SQL, no schema migrations to babysit.** Here's the path from an empty
folder to querying your own tables.
**1. Get the `udb` CLI** — from a release binary, an SDK launcher, or source:
```bash
cargo install --path .
```
**2. Pull in UDB's shared protobuf annotations** so your own protos can use them:
```bash
udb proto export --fmt # writes udb/core/common/v1/*.proto into your project
```
**3. Describe your data model.** Write normal protos and annotate the fields UDB
should turn into tables/columns — that annotation file is the one you just
exported:
```proto
import "udb/core/common/v1/db.proto"; // gives you pg_table / pg_column, RLS, tenancy
```
**4. See exactly what UDB will build** before you run anything — the catalog, the
SQL it generates, and a human-readable lint of your model:
```bash
udb catalog proto # the tables/columns UDB derived from your protos
udb sql proto # the DDL it will run
udb lint proto --human # plain-English warnings (missing tenant column, etc.)
```
**5. Start the broker.** It generates the schema and serves your model over gRPC:
```bash
udb serve proto "" 0.0.0.0:50051
```
**6. Confirm it's healthy:**
```bash
udb doctor --human # readiness in plain English
udb compat-matrix # which backends/features are live
```
**7. Connect your application.** Now point an SDK at the broker, authenticate, and
do tenant-scoped CRUD. The shortest correct path — including the tenant-code →
canonical-UUID handling that trips up most first integrations — is in
[`examples/go_enterprise`](examples/go_enterprise) (Go), with equivalents in
[`examples/python_enterprise`](examples/python_enterprise) and
[`examples/ts_enterprise`](examples/ts_enterprise). Start there rather than
wiring the raw gRPC clients yourself.
## CLI
All public commands use the short binary name `udb`.
| `udb init` | Build or preview a project-aware UDB setup plan |
| `udb proto export --fmt` | Export UDB annotation and broker protos into an app |
| `udb proto fmt --check` | Check annotation formatting |
| `udb catalog`, `udb sql`, `udb plan`, `udb drift` | Inspect schemas, SQL, migrations, and drift |
| `udb sdk generate` | Generate descriptor-aware SDK surfaces from templates |
| `udb native list`, `udb native manifest`, `udb native docs` | Inspect native-service contracts |
| `udb app init` | Add a language/framework integration scaffold |
| `udb dev up`, `udb dev smoke` | Run and test the local playground |
| `udb serve` | Start the gRPC broker |
| `udb doctor`, `udb health-check` | Check runtime and dependency readiness |
| `udb auth ...` | Use control-plane helpers for principals, sessions, API keys, roles, relations, and policies |
| `udb dbops sync` | Sync generated database-operation artifacts |
## Backends
UDB knows these backend kinds:
`postgres`, `mysql`, `sqlite`, `sqlserver`, `clickhouse`, `redis`, `memcached`,
`qdrant`, `weaviate`, `pinecone`, `minio`, `s3`, `azureblob`, `gcs`, `mongodb`,
`elasticsearch`, `neo4j`, and `cassandra`.
The live capability matrix distinguishes compiled support from configured
runtime availability:
```bash
udb compat-matrix
```
## Native Control Plane
UDB ships a separate internal control-plane listener for native services. Bind
it to an internal interface or place it behind a trusted gateway.
| Authn | Login, sessions, refresh tokens, JWT/JWKS, MFA, OTP, devices, WebAuthn |
| Authz | RBAC, ABAC, ReBAC, decisions, policy bundles, native access, governance |
| API keys | Hashed keys, scopes, rotation, revocation, usage stats |
| Identity providers | OIDC, SAML, SCIM, JIT, external identity links |
| Control distribution | Versioned policy/resource distribution with ACK/NACK |
| Tenant, notification, analytics | Tenant config, messages, templates, metrics, SLA views |
| Storage and asset | Object metadata, presigned URLs, asset pipelines, vector-ready workflows |
| WebRTC | Rooms, peers, tracks, TURN credentials, signaling, egress/SFU |
| Vault | Secrets, transit encrypt/sign/HMAC, dynamic database credentials |
| Metering | Usage recording and per-tenant quotas |
| Scheduler, workflow, lock | Cron/one-shot jobs, durable sagas, distributed locks with fencing tokens |
| Search, embedding | Managed search indexes, vector retrieval, and backfill enumeration |
| Webhook | SSRF-guarded outbound webhook endpoints and delivery tracking |
| Config, live query | Feature flags and server-streaming live-query subscriptions |
| Backup | Tenant-scoped backup and restore |
Details, including the read-your-writes consistency contract (write receipts,
read fences, consistency modes) and idempotency `was_duplicate` replay:
[docs/native-services.md](docs/native-services.md).
## SDKs
| Go | `go get github.com/fahara02/udb/sdk/go@v0.4.13` |
| Python | `pip install udb-client==0.4.13` |
| TypeScript / Node | `npm i @udb_plus/sdk@0.4.13` |
| PHP / Laravel | `composer require fahara02/udb-laravel:^0.4.13` |
| C# | `dotnet add package Udb.Client --version 0.4.13` |
| Java | `dev.udb:udb-java-client` (`0.4.13` target; build from checkout until publishing lands) |
Start here: [sdk/README.md](sdk/README.md).
## Examples
| [examples/go_arbitary_project](examples/go_arbitary_project/README.md) | Go app with app-owned protos and UDB broker calls |
| [examples/python_arbitary_project](examples/python_arbitary_project/README.md) | Python app flow with generated models and SDK calls |
| [examples/php_arbitary_project](examples/php_arbitary_project/README.md) | PHP/Laravel app flow |
| [examples/php_quickstart](examples/php_quickstart/README.md) | CRUD, authz scopes, and relationships |
| [examples/native-services](examples/native-services/README.md) | Native auth/authz/API-key/native-access examples |
| [examples/multi_project](examples/multi_project/README.md) | One broker serving separate project catalogs |
| [examples/toy_backend_plugin](examples/toy_backend_plugin/README.md) | Minimal external backend plugin |
## Documentation
- [docs/README.md](docs/README.md) - documentation index
- [docs/architecture.md](docs/architecture.md) - architecture, request flow, routing, pooling, events, and backend capability
- [docs/api-rules.md](docs/api-rules.md) - API route rules, OpenAPI operation IDs, and SDK alias policy
- [docs/annotations.md](docs/annotations.md) - proto annotations
- [docs/integration.md](docs/integration.md) - application integration
- [docs/native-services.md](docs/native-services.md) - native auth, authz, IdP, storage, assets, WebRTC, and SDK facades
- [docs/operations.md](docs/operations.md) - production readiness, config, runbooks, SLOs, and validation
- [docs/security.md](docs/security.md) - request context, identity, authorization, sensitive data, and compliance profiles
- [docs/testing.md](docs/testing.md) - test commands and live checks
- [VERSIONING.md](VERSIONING.md) - release and protocol versioning
- [TESTING.md](TESTING.md) - top-level test guide
## Development
Fast local checks:
```bash
cargo test --lib
buf lint
buf build
node scripts/check-versions.mjs
```
SDK conformance:
```bash
node sdk-conformance/run.mjs
```
The default Rust suite is designed to run without external services. Live
backend, HA, and load checks are documented separately and require the matching
infrastructure.
## License
MIT. See [LICENSE](LICENSE).