# nopace
`nopace` is an asynchronous admission gate for rate-limited APIs. It decides
**when a physical provider attempt may start**, charges every budget that
attempt consumes, preserves capacity for important work, and learns stricter
pacing from provider responses.
It is intended for APIs such as GitHub, Meta, and Google Ads, where one request
can consume several overlapping limits and where being a good citizen matters
more than maximizing request volume.
```text
Call -> acquire all claims atomically -> Permit -> provider attempt -> Feedback -> finish
| |
+-- Wait until safe ------+-- quota is already consumed
```
`nopace` does not own HTTP, serialization, retries, or circuit breaking. This
boundary is deliberate:
- your HTTP client still constructs and sends requests;
- your retry library still decides whether an error is retryable;
- every retry re-enters `nopace`, so every physical attempt consumes quota;
- a circuit breaker handles provider health, while `nopace` handles provider
capacity.
## Status
The current implementation includes the hardened core protocol and four store
backends:
| Fixed windows, including UTC day/month | Implemented |
| Exact sliding windows | Implemented |
| Token buckets | Implemented |
| Leased concurrency | Implemented |
| Atomic multi-claim admission | Implemented |
| Niceness, reserved headroom, and headroom snapshots | Implemented |
| Remaining/reset, not-before, and temporary-throttle feedback | Implemented |
| Closure, explicit permit, and typed request APIs | Implemented |
| Deterministic manual clock and contention tests | Implemented |
| Memory store | Implemented; one process only |
| PostgreSQL store | Implemented; transactional scope-row serialization |
| Redis store | Implemented; fenced atomic commits and server time |
| NATS store | Implemented; JetStream KV compare-and-swap, one key per partition |
| Application-defined `Store` implementations | Implemented; public extension contract |
Use the memory store when all callers sharing a budget live in one process.
Use PostgreSQL, Redis, NATS JetStream, or an application-owned registry for
multiple replicas. Every backend runs the same replay, completion, renewal, and
atomic admission contract, with one documented exception: NATS keys durable
state per partition, so a call whose claims span several partitions is refused
rather than admitted non-atomically. See
[`ScopeKey`: the atomicity boundary](#scopekey-the-atomicity-boundary).
## Install
```toml
[dependencies]
nopace = "0.1"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
```
The default `memory` feature re-exports `MemoryStore`. Core types can be used
without a backend:
```toml
nopace = { version = "0.1", default-features = false }
```
## Quick start
A token bucket allows an initial burst of two calls, then refills one unit every
100 ms:
```rust
use std::time::Duration;
use nopace::{Call, Claim, Limit, MemoryStore, Pace, Policy};
# async fn example() -> Result<(), Box<dyn std::error::Error>> {
let pace = Pace::builder(MemoryStore::new())
.scope("github-token")?
.policy(
Policy::new("github-core")?.limit(
Limit::token_bucket("burst", 2, 1, Duration::from_millis(100))?
),
)
.build()?;
let call = Call::new("list-issues")?
.claim(Claim::new("primary", "github-core", "token-fingerprint")?);
let issues = pace
.run(call, || async {
// The provider request starts only after admission.
Ok::<_, std::convert::Infallible>(vec!["issue-1", "issue-2"])
})
.await?;
assert_eq!(issues.len(), 2);
# Ok(())
# }
```
Construct one `Pace` at application startup and clone it into handlers. Cloning
a `Pace` or `MemoryStore` shares the same state; constructing another
`MemoryStore::new()` creates independent state.
## The design in six types
A successful integration starts by modeling the provider's quota vocabulary
rather than its HTTP endpoints.
### `ScopeKey`: the atomicity boundary
A scope is configured once on `Pace`. All claims made through that gate are
evaluated and committed together.
Choose the broadest stable identity across which budgets can overlap. Examples:
- a GitHub credential fingerprint when primary and secondary limits share that
credential;
- a Google Ads developer-token fingerprint when calls also consume customer and
endpoint budgets;
- a Meta application ID when app, ad-account, and user pressure overlap.
**On NATS, the partition is the atomicity boundary, not the scope.** JetStream
KV gives compare-and-set on one key and no transaction across keys, and that
store keys durable state per `(scope, partition)` so one tenant's value cannot
grow with the tenant count. The last two examples above — several partitions in
one call — are therefore refused by `NatsStore` with
`NatsError::CrossPartitionCall`, loudly, rather than split into decisions that
would each let the other's traffic through. If a call genuinely spans
partitions, either give the group its own partition key or use PostgreSQL or
Redis, whose durable unit is the scope. Nothing changes for the common case of
one claim, or of several claims against one partition.
Do not create a scope per request. Do not put raw credentials in a scope or
partition. Use a stable cryptographic fingerprint, ideally an HMAC with an
application-owned key.
### `Policy`: one provider budget family
A policy is a named, immutable collection of limits plus optional priority
headroom. For example, `github-core` might contain an hourly fixed window, a
secondary rolling-minute window, and a concurrency bound.
A claim must satisfy **every limit in its policy**. Keep policy and limit IDs
stable across deployments; changing their meaning while retaining their names
would reinterpret existing state in a future persistent backend.
### `PartitionKey`: who shares a budget
The partition is the runtime identity charged by a claim:
- credential fingerprint;
- application ID;
- customer or ad-account ID;
- authenticated user ID;
- repository or endpoint group, if the provider defines such a budget.
Equal `(policy, partition)` pairs share state. Different partitions are
independent. Partition only where the provider partitions: inventing per-user
partitions for a credential-wide provider limit would multiply capacity and
over-admit.
### `Claim`: one budget consumed by an attempt
A claim connects a stable label, policy, partition, and integer cost:
```rust
# use nopace::{Call, Claim, Niceness};
# fn call(token: &str, customer: &str) -> Result<Call, nopace::ValidationError> {
Ok(
Call::new("google-ads-search")?
.claim(
Claim::new("developer", "developer-daily", token)?.units(5)?
)
.claim(
Claim::new("customer", "customer-minute", customer)?
)
.niceness(Niceness::NORMAL)
)
# }
```
One physical call can have several claims. Admission is all-or-none: if the
customer budget is exhausted, the developer budget is not partially consumed.
Costs are quota units, not necessarily request counts; a batch mutation can
charge the number of operations it contains.
Claim labels are local to the call and let response feedback target the admitted
state safely. Use unique labels within a call.
Identifiers are 1-128 bytes and may contain ASCII letters, digits, and
`-._:/`. This applies to call names, claim labels, policy IDs, limit IDs,
partitions, and scopes.
### `Call`: one physical provider attempt
A call describes the attempt before it runs:
- a diagnostic name;
- all quota claims;
- niceness from `0` (important) to `19` (background);
- an optional `Instant` deadline.
Create a fresh `Call` for every physical retry. `nopace` does not retry closures
and does not decide whether a failed mutation is safe to repeat.
### `Permit`: proof of admission
Quota is consumed when the permit is granted, before provider code starts.
`finish` applies response feedback and releases concurrency leases.
Dropping an unfinished permit is intentionally conservative:
- fixed/sliding/token quota remains consumed because the provider may have seen
the request;
- concurrency remains occupied until its lease expires;
- there is no async refund from `Drop`.
Always drive permits to `finish`, including provider failures, whenever the
process is still alive.
## Limit algorithms
Choose the algorithm that matches the provider contract. A more expensive
algorithm is not inherently safer if it models the wrong boundary.
| `FixedWindow` | The provider resets at deterministic boundaries | Integer usage within epoch-aligned durations, UTC calendar days, or UTC calendar months. Cheap and compact. |
| `SlidingWindow` | The provider says “the preceding N seconds/minutes” | Stores exact admitted `(timestamp, units)` entries. Exact rolling semantics, but more state and work. |
| `TokenBucket` | The provider allows bursts with a steady refill rate | Integer capacity and refill. Smooth pacing without floating-point drift. |
| `Concurrency` | The provider caps simultaneous in-flight requests | A leased slot released by `finish`, recoverable after crashes through TTL. |
Several horizons belong in the same policy when they constrain the same
partition:
```rust
# use std::time::Duration;
# use nopace::{Limit, Policy, Window};
# fn policy() -> Result<Policy, nopace::PolicyError> {
Ok(
Policy::new("provider-core")?
.limit(Limit::fixed_window("daily", 100_000, Window::utc_day())?)
.limit(Limit::sliding_window("rolling-minute", 1_000, Duration::from_secs(60))?)
.limit(Limit::token_bucket("burst", 20, 1, Duration::from_millis(100))?)
.limit(Limit::concurrency("in-flight", 10, Duration::from_secs(30))?)
)
# }
```
### Concurrency lease safety
A concurrency limit is a crash-recoverable bound, not a distributed mutex. If a
process cannot renew and its request outlives the lease, another request may be
admitted while the first is still running.
Current renewal is explicit:
```rust
# use std::time::Duration;
# use nopace::{Call, Feedback, Pace, Store};
# async fn request<S: Store>(pace: &Pace<S>, call: Call) -> Result<(), nopace::AcquireError<S::Error>> {
let mut permit = pace.acquire(call).await?;
let renewed = permit.renew(Duration::from_secs(30)).await;
// Execute the provider request, renewing periodically if it can outlive the lease.
// Preserve the permit until the operation completes.
let _ = renewed;
let _ = permit.finish(Feedback::none()).await;
# Ok(())
# }
```
Set `lease_ttl` above the normal request duration and renew well before expiry.
Use the explicit permit API for long operations; `Pace::run` does not currently
renew automatically. For a hard side-effect mutex, use provider idempotency keys
or a database lock in addition to `nopace`.
## Priority without a distributed scheduler
`Niceness` follows Unix intuition: `0` is most important, `19` is background.
Priority policies reserve capacity for more important work; `reserve(threshold,
basis_points)` withholds that fraction from calls at that niceness and worse
rather than granting that fraction to them.
```rust
# use nopace::{BasisPoints, Niceness, PriorityPolicy};
# fn priority() -> Result<PriorityPolicy, nopace::PolicyError> {
PriorityPolicy::builder()
.reserve(Niceness::NORMAL, BasisPoints::new(2_000)?)
.reserve(Niceness::BACKGROUND, BasisPoints::new(5_000)?)
.build()
# }
```
This means:
- important calls can use the full provider capacity;
- normal calls stop after consuming their 80% usable share;
- background calls stop after consuming their 50% usable share;
- every admitted call still counts against the same hard provider limit;
- priority never increases provider capacity.
Reservations are integer basis points (`10_000 = 100%`) and must increase as
niceness increases. This preserves headroom but does not guarantee FIFO ordering
or prevent starvation caused by an unsuitable policy. Use
`PriorityPolicy::reservations`, `reservation`, or `usable_capacity` when tests
need to assert the configured policy directly. `nopace` is an admission gate,
not a distributed work scheduler.
## Provider feedback
Static policy is the conservative starting point. Response feedback may make the
next admission stricter based on what the provider actually reported.
`FeedbackBuilder` is created from a permit and accepts only claim/limit labels
that were admitted. This catches parser mistakes before unrelated state is
updated.
### Remaining and reset
Use exact remaining quota and an absolute Unix-millisecond reset timestamp:
```rust
# use nopace::{ClaimLabel, Feedback, FeedbackBuilder, LimitId};
# fn parse(feedback: FeedbackBuilder) -> Result<Feedback, Box<dyn std::error::Error>> {
let claim = ClaimLabel::new("primary")?;
let limit = LimitId::new("hourly")?;
Ok(feedback
.remaining(&claim, &limit, 42, 1_800_000_000_000)?
.build())
# }
```
Reported remaining is a ceiling on local admission. It cannot increase the
configured static capacity.
### Not before
Use `not_before` for `Retry-After`, explicit provider reset errors, or a
documented recovery time. Target one claim, or pass `None` to block every claim
carried by that permit.
### Temporary throttle
Use `temporary_throttle` for provider pressure telemetry such as Meta usage
percentages. It overlays a temporary token bucket with a minimum interval,
burst, and expiry; once expired, the static policy remains.
Keep parsing provider-specific data outside `nopace`. The core deals only in
transport-neutral signals, which keeps GitHub headers, Meta JSON headers, and
Google Ads error payloads from becoming core dependencies.
### Runtime headroom
`Pace::headroom(call)` returns the current headroom for every limit that would
gate that call, including learned provider constraints. The result is a snapshot,
not a reservation; admission can still change before the next `acquire`.
## Three execution styles
| Plain provider call; finish with no response feedback | `run` |
| Provider headers/body update pacing state | `run_observed` |
| Manual renewal, split parsing, or custom cleanup | `acquire` / `Permit` |
| Queue or scheduler owns waiting | `try_acquire` |
| Typed SDK wrapper | `Request` / `execute` |
| Dashboard or preflight budget snapshot | `headroom` |
### Closure API: default for ordinary HTTP calls
`run_observed` keeps admission adjacent to the provider call and persists feedback
from both success and failure:
```rust
# use nopace::{Call, Observed, Pace, Store};
# async fn send<S: Store>(pace: &Pace<S>, call: Call) {
let result = pace
.run_observed(call, |feedback| async move {
// let response = client.send().await;
// Parse provider headers even when the status is 429/403.
let provider_result: Result<&str, &str> = Ok("response");
match provider_result {
Ok(value) => Ok(Observed::new(value).feedback(feedback.build())),
Err(error) => Err(Observed::new(error).feedback(feedback.build())),
}
})
.await;
# let _ = result;
# }
```
The closure is `FnOnce`; `nopace` never reruns it.
### Explicit permit API: maximum control
Use `acquire`/`finish` when:
- provider output and feedback-store failure need separate handling;
- concurrency needs periodic renewal;
- response parsing spans several layers;
- cleanup must be coordinated explicitly.
`RunError::Operation(error)` displays the provider error, and
`into_operation_error()` extracts it when the retry policy wants the original
type. `RunError::Feedback { outcome, source }` is important: the provider
attempt already happened, and the complete success or failure outcome is
preserved even though feedback persistence failed. Do not blindly repeat a
mutation in this case.
### One-shot admission: external queues and schedulers
Use `try_acquire` when another component owns waiting:
```rust
# use nopace::{Admission, Call, Pace, Store};
# async fn schedule<S: Store>(pace: &Pace<S>, call: Call) -> Result<(), nopace::AcquireError<S::Error>> {
match pace.try_acquire(call).await? {
Admission::Granted(permit) => {
// Dispatch now, retaining the permit for finish.
let _ = permit;
}
Admission::Wait { retry_at, limited_by } => {
// Re-enqueue for retry_at.retry_at(); limited_by explains every failing constraint.
let _ = (retry_at, limited_by);
}
}
# Ok(())
# }
```
`retry_at` is an eligibility hint. Fixed windows, sliding windows, token
buckets, and learned reset times return `Eligibility::At(_)`; saturated
concurrency returns `Eligibility::NoLaterThan(_)` because a permit may finish
before its lease TTL. `acquire` polls upper-bound hints using
`PaceBuilder::uncertain_wait_interval`.
Typed SDK wrappers can implement `Request` and use `Pace::execute`; the
request's `call()` remains visible and reviewable.
## Put `nopace` inside the retry loop
The safe composition is:
```text
retry policy
-> nopace acquire
-> one physical HTTP attempt
-> parse and persist feedback
-> classify result and maybe retry
```
Never acquire once around an entire retry loop. That would charge several
physical attempts as one and delay `Retry-After` feedback until retries are
already complete.
On a throttling failure:
1. parse `Retry-After`, remaining/reset, or provider pressure;
2. return `Observed::new(error).feedback(feedback)`;
3. let `run_observed` finish the permit and persist the signal;
4. only then let the retry policy decide whether another attempt is safe;
5. the next attempt re-enters `nopace` and waits on the learned constraint.
Circuit breaking remains separate. A circuit breaker answers “is this dependency
healthy enough to try?”; `nopace` answers “would this attempt respect its
budgets?” A typical stack checks the circuit breaker before waiting for
admission, then records the provider outcome after the attempt.
## Building a successful policy
Use this sequence for each provider integration.
### 1. Inventory authoritative limits
Record for every documented limit:
- identity it applies to: credential, app, customer, user, endpoint;
- capacity and quota unit;
- fixed, calendar, rolling, refill, or concurrency semantics;
- reset source and timestamp format;
- response/error fields that reveal stricter state;
- whether different endpoint costs exist.
Do not assume one request equals one unit.
### 2. Select one broad scope
Choose a scope containing every budget a physical call can touch. All processes
sharing those budgets must use the same scope and the same store. A narrower
scope duplicates capacity; an unnecessarily global scope adds contention.
### 3. Define stable policies and partitions
Name policies after provider budget families, not application functions. Derive
partitions only from identities the provider actually isolates. Fingerprint
credentials; never log or store raw secrets.
### 4. Use conservative static limits
Bootstrap from documented limits and leave safety margin where the provider is
approximate or where other uncoordinated consumers exist. Feedback can tighten
admission after a response, but the first call still needs a safe bound.
### 5. Attach every claim atomically
Describe every budget consumed by the physical attempt before it starts. If an
endpoint consumes developer-daily, customer-minute, endpoint-QPS, and
concurrency, include all four claims/policies in one `Call`.
### 6. Parse both success and failure responses
Rate information often arrives on successful responses; recovery times often
arrive on `429`, `403`, or provider-specific quota errors. Use `run_observed`
when preserving feedback on `Observed<T>` and `Observed<E>` is required.
### 7. Set deadlines and choose waiting ownership
Use `Call::deadline(Instant)` when a request becomes worthless after a point.
Use `acquire` when the task may wait in-process; use `try_acquire` when a
durable queue should own the delay. `try_acquire` exposes exact eligibility
times and upper-bound concurrency hints separately.
`stores::ManualClock` makes resets, rolling expiry, refill, feedback blocks, and
lease expiry deterministic. To test `acquire`/`run` waiting with it, pass the
same clock to both `MemoryStore::with_clock(clock.clone())` and
`PaceBuilder::clock(clock.clone())`; otherwise only store decisions use manual
time. Tests should prove observable invariants:
- the N+1 call waits at the exact boundary;
- a multi-claim rejection changes no budget;
- important traffic can use reserved headroom;
- provider remaining `0` blocks until reset;
- retries consume one unit per physical attempt;
- dropped concurrency permits recover only after TTL;
- different provider partitions do not accidentally share state;
- identities that should share a budget really do.
### 8. Plan store failure semantics
With the memory store, admission is infallible but state disappears on process
restart. Future distributed stores will return operational failures. Treat an
admission-store failure as fail-closed unless violating provider limits is
explicitly safer than delaying work. Treat feedback-store failure as an
ambiguous completed provider operation, not as proof the provider call failed.
## Bring your own store
`Store` is a public extension point. Applications can implement it for an
internal HTTP/gRPC registry, an existing database service, or another
coordination system:
```rust
# use nopace::{AdmissionRequest, Completion, CompletionResult, LeaseRenewal, LeaseRenewalResult, Store, StoreDecision, StorePeek};
# #[derive(Clone)] struct RegistryClient;
# #[derive(Debug)] struct RegistryError;
# impl std::fmt::Display for RegistryError {
# fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { f.write_str("registry unavailable") }
# }
# impl std::error::Error for RegistryError {}
# impl RegistryClient {
# async fn admit(&self, _: AdmissionRequest) -> Result<StoreDecision, RegistryError> { Err(RegistryError) }
# async fn peek(&self, _: AdmissionRequest) -> Result<StorePeek, RegistryError> { Err(RegistryError) }
# async fn renew(&self, _: LeaseRenewal) -> Result<LeaseRenewalResult, RegistryError> { Err(RegistryError) }
# async fn finish(&self, _: Completion) -> Result<CompletionResult, RegistryError> { Err(RegistryError) }
# }
#[derive(Clone)]
struct RegistryStore {
client: RegistryClient,
}
impl Store for RegistryStore {
type Error = RegistryError;
async fn decide(
&self,
request: AdmissionRequest,
) -> Result<StoreDecision, Self::Error> {
self.client.admit(request).await
}
async fn peek(&self, request: AdmissionRequest) -> Result<StorePeek, Self::Error> {
self.client.peek(request).await
}
async fn renew(&self, renewal: LeaseRenewal) -> Result<LeaseRenewalResult, Self::Error> {
self.client.renew(renewal).await
}
async fn finish(&self, completion: Completion) -> Result<CompletionResult, Self::Error> {
self.client.finish(completion).await
}
}
```
The store type must be cheap to clone. Wrap a shared client in `Arc` when that is
the natural ownership model; `Arc<S>` implements `Store` whenever `S` does:
```rust
# use std::sync::Arc;
# use std::time::Duration;
# use nopace::{Limit, Pace, Policy, Store};
# fn gate<S: Store>(store: S) -> Result<Pace<Arc<S>>, nopace::BuildError> {
let store = Arc::new(store);
Pace::builder(store)
.scope("provider-credential")?
.policy(
Policy::new("provider-core")?.limit(
Limit::token_bucket("burst", 10, 1, Duration::from_secs(1))?
),
)
.build()
# }
```
This accepts an `Arc`-wrapped custom store without an adapter crate.
### Store semantics are the safety boundary
Implementing the Rust trait is the small part. The remote service must preserve
the admission contract:
1. `decide` is linearizable over the state the command addresses: the scope, or
the scope and partition for a store that keys state per partition.
2. Every claim in one request is checked and committed atomically. A store whose
durable unit is narrower than the call must refuse the call, not split it.
3. A grant consumes static quota, learned remaining, temporary throttles, and
concurrency in one transaction.
4. Repeating the same admission ID returns the same decision; it never charges
twice.
5. `finish` is idempotent: duplicate delivery applies feedback once and releases
concurrency once.
6. `renew` extends only a live lease belonging to the permit and is safe to
repeat.
7. Store/server time determines fixed boundaries, refill, feedback expiry, and
lease expiry. Do not trust individual application clocks in a distributed
deployment.
8. A rejected multi-claim request changes no quota state.
9. Unknown schema or algorithm versions fail closed rather than resetting quota.
10. State and deduplication records live long enough for all active windows,
throttles, leases, and transport retries.
`AdmissionRequest` contains resolved policies because the store is responsible
for executing their semantics atomically. A thin key/value registry is not
enough: a read followed by an application-side write races under contention.
Put the decision state machine in the authoritative registry, transaction, or
server-side atomic function.
For an API registry, use three idempotent commands:
```text
POST /v1/admissions AdmissionRequest -> StoreDecision
POST /v1/leases/renew LeaseRenewal -> { "renewed": bool }
POST /v1/completions Completion -> { "finished": true }
```
Retry a command with the same payload and ID after timeouts. Do not manufacture a
new admission ID to resolve an ambiguous response: the first request may already
have committed. Return store outages as errors; `Pace` fails closed instead of
silently bypassing the gate.
The runnable `custom-store` example starts a real Axum server on an ephemeral
loopback port, builds a Reqwest-backed `RegistryStore`, and sends admission,
lease renewal, and completion as JSON over HTTP:
```bash
cargo run -p custom-store
```
The server owns a map of `ScopeState` values and executes `decide_at`, `renew_at`,
and `finish_at` while holding one mutex across each complete state transition.
That makes the example a correct single-process registry rather than a
client-side read/modify/write simulation. Its integration tests also replay the
same admission ID over HTTP and prove that the result is idempotent.
The HTTP wire format is enabled by `Serialize`/`Deserialize` implementations on
the public store command and policy types. `Call::deadline` is intentionally
process-local and is not serialized; the client enforces it while waiting, and
the registry receives only the quota model needed for a decision.
The example server is not durable and loses state when it exits. A production
registry must persist the same atomic transitions at its authoritative storage
layer and define authentication, authorization, request-size limits, schema
versioning, retention, and high availability. Before deployment, run contention,
duplicate delivery, timeout-after-commit, multi-claim rollback, feedback,
renewal, and TTL recovery tests against the real service.
## Provider examples
The `real-apis` workspace example contains provider-specific adapters and
runnable scenarios:
```bash
# Anonymous or GITHUB_TOKEN + GITHUB_USER_ID
cargo run -p real-apis -- github
# META_ACCESS_TOKEN, META_APP_ID, META_AD_ACCOUNT_ID, META_USER_ID
cargo run -p real-apis -- meta
# GOOGLE_ADS_URL, GOOGLE_ADS_BEARER_TOKEN, GOOGLE_ADS_DEVELOPER_TOKEN,
# GOOGLE_ADS_CUSTOMER_ID, GOOGLE_ADS_USER_ID
cargo run -p real-apis -- google-ads
# Local demonstration of user/customer/credential partitioning
cargo run -p real-apis -- partitions
```
What they demonstrate:
| GitHub | credential hourly quota, secondary rolling points, concurrency, priority headroom | `x-ratelimit-remaining`, `x-ratelimit-reset`, `Retry-After` |
| Meta | app rolling budget, app/ad-account/user partitions, concurrency | `x-app-usage`, `x-business-use-case-usage`, recovery time |
| Google Ads | developer daily units, customer minute budget, optional endpoint QPS/concurrency | `RESOURCE_EXHAUSTED`, concurrent-request errors, `Retry-After` |
Provider limits change. Treat these as integration patterns, not a substitute
for checking current provider documentation and limits assigned to your account.
Smaller examples:
```bash
cargo run -p basic-memory
cargo run -p priority-headroom
```
## Operational checklist
Before production:
- [ ] One long-lived `Pace`/store is shared by all local handlers.
- [ ] Every process consuming the same provider budget coordinates through one
admission owner; memory state is not mistaken for distributed state.
- [ ] Scope is broad enough for every overlapping claim.
- [ ] Raw tokens and credentials never appear in scope, partition, logs, or
metrics.
- [ ] Policy/limit IDs and partition derivation are stable.
- [ ] Static limits are conservative and use correct quota units.
- [ ] Every physical retry creates a fresh call and re-enters the gate.
- [ ] Success and throttling responses both emit feedback.
- [ ] Long operations use adequate concurrency TTL and explicit renewal.
- [ ] `RunError::Feedback` is handled as a completed provider attempt.
- [ ] Deadlines prevent obsolete work from waiting indefinitely.
- [ ] Deterministic boundary and contention tests cover the actual provider
model.
## Crates and features
- `nopace`: facade crate and the recommended application dependency.
- `nopace-core`: transport-independent model, algorithms, execution API, and
`Store` contract.
- `nopace-memory`: cloneable process-local store with system and manual clocks.
- `nopace-postgres`: transactional durable PostgreSQL store. It owns one
`tokio_postgres::Client` and serializes operations through it; services that
need an existing pool should wrap their own `Store`.
- `nopace-redis`: atomic Redis store with fenced operation commits.
- `nopace-nats`: durable NATS JetStream KV store. One compare-and-set key per
`(scope, partition)`, so a value's size follows one tenant's traffic rather
than the tenant count, and unrelated tenants never share a revision. A delete
or purge tombstone resets a partition. Because two partitions are two keys
with no shared transaction, a call whose claims span several partitions is
refused rather than split; see [`DESIGN.md`](DESIGN.md) section 6.4.
Facade features:
| `memory` | yes | Re-exports `MemoryStore` and clock types |
| `postgres` | no | Re-exports transactional `PostgresStore` |
| `redis` | no | Re-exports fenced atomic `RedisStore` |
| `nats` | no | Re-exports JetStream KV `NatsStore` |
See [`DESIGN.md`](DESIGN.md) for the distributed-store contracts, backend
atomicity strategy, and acceptance matrix.