Keel
The SQLite of durable execution. Production-grade resilience and crash-resumable workflows, running inside your process — no service to deploy, no database to provision, no code to rewrite.
$ keel run app.py
keel ▸ wrapped 14 call sites (httpx ×9, openai ×4, psycopg ×1) with production defaults — `keel init` to customize
That's it. Your outbound HTTP calls, database queries, and LLM requests now retry on transient failures, back off exponentially, trip a circuit breaker under sustained failure, respect rate limits, and — if you opt in — survive a crash and resume exactly where they left off. Your code is untouched. Uninstall the package and you're back to exactly what you had before.
The problem
Every service that talks to the network eventually gets paged for the same handful of reasons: a downstream dependency blipped and nothing retried it, a retry storm took down a service that would have recovered on its own, or a long-running job died halfway through and had to restart from zero. Fixing this yourself means scattering retry decorators across a codebase and hoping every new call site remembers them. Fixing it "properly" usually means adopting a workflow engine — a service to run, a database to provision, and a rewrite of business logic into activities and workflows — for problems that were never distributed-systems-scale to begin with.
Keel takes neither path. It's a library, not a service: it intercepts calls
you're already making, applies policy from one keel.toml file, and — only
when a call needs to survive a crash — journals it to a local file. No
daemon. No port. No new abstractions in your code.
Why Keel
| Hand-rolled retry decorators | Workflow engines (Temporal-style) | Keel | |
|---|---|---|---|
| Code changes | Decorate every call site | Rewrite as activities/workflows | Zero |
| Infrastructure | None | A service, a database, a cluster | None — a local file |
| Consistency across a codebase | Whatever each engineer remembers | Enforced by the framework | Enforced by one policy file |
| Crash-resumable execution | No | Yes | Opt-in, same library |
| Removing it | Undo the decorators, one by one | A migration project | Uninstall the package |
What you get
- Zero code changes. Keel patches the interception seams your language already exposes (Python's import hooks, Node's ESM loader, an attribute macro for Rust) — your source is never touched, and uninstalling the package restores your original behavior exactly.
- Production-grade defaults, out of the box. Every discovered outbound call gets a 30s timeout, 3 retries with jittered exponential backoff on transient errors, and a per-host circuit breaker — before you write a single line of config.
- One
keel.toml, not a decorator per call site. Retry schedules, timeouts, rate limits, caching, circuit breakers, and poll-until-terminal submit-then-poll loops are policy, not code — reviewed like infrastructure, not scattered through business logic. - Opt-in durable execution. Designate a function as a flow and its steps
are journaled:
kill -9it mid-run, and rerunning it replays completed steps from the journal instead of re-executing their side effects — proven by real subprocess crash-and-resume tests, not a mocked clock. - Observable when you need it, invisible when you don't. OpenTelemetry spans and metrics for every call and attempt are one build feature and one env var away — off by default, so the shipped library carries no OpenTelemetry dependency until you ask for it.
- Built for LLM and agent workloads. First-class
llm:/tool:/mcp:targets, per-run spend caps, model fallback chains, and a dev-mode cache that replays identical prompts for free — because agent code is the densest concentration of flaky, expensive effects in modern software. - Fast enough to be invisible. The wrapped-call path measures ~0.8µs worst case against a 10µs budget — resilience you can't feel.
- Agent-native tooling.
keel mcpserves the CLI itself as an MCP server; every command has a deterministic--jsontwin;keel explain <code>gives a coding agent the exact remedy without a web search. - Two languages today, checked against each other. Python and
Node/TypeScript both run on the same real, tested Rust core; a scoped
Rust front end covers
#[keel::wrap]-annotated functions directly. Every implementation is checked against the same conformance suite — the tests are the spec, not the docs.
Quickstart
Keel is published today — pip, npm, and cargo all work.
Python — library only, or library + the keel CLI in one line:
Node (≥ 22.5) — same shape:
Rust — the library (#[keel::wrap]) and the CLI are always separate
installs; cargo add/cargo install are different operations and cargo has
no single command spanning both:
Just want the CLI, no persistent install, any language?
Building from source (contributors — needs Rust; rustup picks up the
pinned toolchain automatically):
Without the native module, the front end falls back to a pure-Python core: Tier 1 resilience still works, but there's no persistent cache and no durable flows.
See it work
Five runnable, deterministic demos — no real network involved
(tools/faultproxy serves scripted faults). See the
40-second storyboard for the shooting script.
| Demo | What it proves | Language |
|---|---|---|
| flaky-python | A bare script dies on a 503; keel run survives it |
Python |
| node-service | Same story, Node: a bare script dies on a 500; keel run survives it |
Node |
| agent-demo | An LLM call survives a 429 storm; a second run costs ~0 API calls (dev cache) | Python |
| adk-demo | A real google-adk agent's tool call survives a 429 storm below the agent loop — zero extra LLM tokens |
Python |
| durable-pipeline | kill -9 mid-flow, rerun, and it resumes 10/10 steps — each firing exactly once |
Python |
How it works
Two tiers, one policy file:
-
Tier 1 — resilience. Every intercepted call passes through a fixed layer chain: cache → rate limit → circuit breaker → timeout → retry. Stateless, works everywhere, needs nothing but the library.
-
Poll (submit-then-poll, opt-in). A
polltable turns a GET/HEAD call into a poll-until-terminal loop, judged on the response body instead of the transport result — the shape behind "submit a job, poll its status field until done":[] = { = "10s", = "90s", = { = "status", = ["completed", "failed"] } }A non-terminal
statuspastdeadlinefails terminally withKEEL-E016; a response whose body isn't JSON (or lacksuntil.field) fails OPEN and is returned unchanged on the first attempt — polling never turns an ordinary response into an error. -
Tier 2 — durable flows (opt-in). Designate an entrypoint in
[flows]and its steps are journaled to a local SQLite file (or Postgres, for fleet deployments) as they run. A crash — or a deliberate restart — replays completed steps from the journal instead of re-firing their side effects, then resumes live from wherever it left off.
keel exec — durable external commands (CCR-4)
Wrap any command as a journaled durable flow — at-most-once dispatch per identity, crash-safe retry gating, and a declared-side-effect gate:
keel exec --flow autonomous-run \
--journal-file logs/trades.jsonl \
-- ./run_autonomous.sh
Honest scope: Keel gives you at-most-once dispatch + crash-safe retry
gating, not exactly-once execution inside an opaque child. A concurrent
same-identity invocation follows [flows] on_busy = "skip" | "wait" | "fail" (default skip — the mkdir-mutex pattern this replaces). If a
failed run's declared journal files changed, a retry is refused with
KEEL-E033 (--force overrides). A completed flow re-invoked with the same
identity replays instantly without respawning the child.
Both tiers run on the same native Rust core via a C ABI, so the Python and Node front ends share identical semantics — verified by a shared conformance suite that every implementation must pass, not just documentation asserting it.
Agent integration
Two ways a coding agent picks up Keel:
- A Claude Code Skill (
packaging/claude-skill/keel/) — covers adopting Keel in a project, day-to-day commands, and drivingkeel mcp. Install it by copying the directory into~/.claude/skills/keel/or a project's.claude/skills/keel/. keel init --agentsdrops a concise, deterministic section intoAGENTS.mdso every future agent session in an already-Keel-adopted repo inherits the ground rules without installing anything extra.
Activation without keel run
When another tool owns the process launch (agents-cli run, adk api_server,
uvicorn, a test runner), Keel can activate as a plain dependency:
- Python — the
keelrunwheel ships a site-packages.pthshim gated on one env var. SetKEEL_ENABLE=1(e.g. in your project.env) and every Python process in that environment boots with the same policy enginekeel runuses —keel.tomlfrom the working directory, or fromKEEL_CWD=<dir>when your config lives in the deployable app directory. - Node — add
NODE_OPTIONS="--import keelrun/register"alongsideKEEL_ENABLE=1.
Activation is fail-open by design: a broken install or invalid keel.toml
prints one keel ▸ warning line and your app runs unwrapped. KEEL_DISABLE=1
always wins. The preflight resilience advisory stays a keel run-only,
CLI-side feature in both languages. Python's .pth shim additionally does
not wire keel record/keel sim or dispatch flow entrypoints — a .pth has
no target script to match [flows] entrypoints against. Node's
keelrun/register is a thin KEEL_ENABLE gate around the same preload
keel run uses — flow-entrypoint dispatch and
KEEL_RECORD/KEEL_SIM_PLAN wiring behave exactly as under keel run.
keel mcp serves the CLI itself as an MCP server over stdio — six tools,
each byte-identical to its --json CLI twin (get_status,
get_doctor_report, propose_policy, get_trace, list_flows,
explain_error). It has no --project flag; it always reports on its own
current working directory. Project-scoped <project>/.mcp.json (Claude
Code — already launched with the right cwd):
Global config (Claude Desktop's claude_desktop_config.json, or any client
that doesn't launch from the project directory) needs an explicit cwd:
No keel on PATH (only installed via uvx)? Swap in
"command": "uvx", "args": ["--from", "keelrun-cli", "keel", "mcp"].
Keel for Google ADK + agents-cli
Three steps, no code changes:
- Dependency —
uv add keelrun(or start from the keel-enabled template:agents-cli create my-agent -a MisterTK/keel/packaging/agents-cli-template). - Activate —
KEEL_ENABLE=1in your project.env(agents-cli propagates it to local runs, eval, and every deploy target). - Policy —
keel initwriteskeel.tomlinto your agent directory (inside the Dockerfile's COPY set, so it ships in the container —keel doctorwarns if it ever ends up at the repo root instead).
Every Gemini call (llm:google-genai), tool call (tool:<name>), and MCP
server round trip (mcp:<server>) becomes a policy-governed Keel target —
certified weekly against the real google-adk and mcp packages in CI,
including a full agent-over-MCP composition test. See demos/adk-demo for
a runnable 429-survival demo, and skills/keel/ (npx skills add MisterTK/keel) for the coding-agent skill.
Two ADK-specific capabilities beyond the target list above, both
farm-certified against the real google-adk package: list
py:google.adk.runners:Runner.run_async under [flows] entrypoints for
durable, crash-resumable agent turns (a designated Runner.run_async call
becomes a Tier 2 flow — see docs/targeting.md for the v1 limitations,
notably one flow per process); and set fallback = [...] on the
llm:google-genai target for cross-model fallback that survives a
provider switch, not just a same-provider retry — the plugin's
on_model_error hook resolves and constructs a real fallback model via
ADK's own LLMRegistry, the one seam that can build a request for a
genuinely different provider.
Status
Keel is pre-1.0 and published on every registry (pip, npm, cargo — see
Quickstart above; the front-end name is keelrun, the CLI is
keelrun-cli, see docs/naming-decision.md). brew install keel is not
available — the Homebrew tap was deliberately not created (cargo/pip/
npm/uvx already cover every platform). Everything
described in this README is
real, tested, and running on the native core in both languages today — this
isn't a roadmap, it's what's built. What's explicitly not built yet: a
zero-config Rust CLI wrapper (Rust requires the #[keel::wrap] attribute
instead), custom regex retry conditions, an object-store-backed journal for
massive scale, and a hermetic/WASM simulation mode.
Bug reports and pull requests are welcome — open an issue or a PR.
Learn more
llms.txt/llms-full.txt— compact, retrieval-friendly docs for coding agents evaluating or integrating Keel.conformance/README.md— the normative behavior every implementation is tested against.contracts/README.md— the frozen interfaces (policy schema, FFI, journal, adapter-pack contract) and how they change.python/keel/README.md/node/keel/README.md— full front-end reference for each language.
Licensed under Apache-2.0.