Expand description
Watchfire, the agent runtime of the Smeltery
framework: supervised long-running agents, background jobs on a queue, and a scheduler, on
one Tokio runtime. Apps use it as smeltery::watchfire.
§smeltery-watchfire
Watchfire is the agent runtime of the Smeltery framework: supervised
long-running agents, background jobs on a queue, and a scheduler, on one Tokio runtime. Apps use it
through the facade as smeltery::watchfire; the full guide is the “Watchfire” section of the Smeltery README.
use smeltery_watchfire::prelude::*;
pub fn register(w: &mut Watchfire) {
w.every(30.secs(), "heartbeat", |ctx| async move {
ctx.log().info("tick");
Ok(())
});
w.run("scraper", |ctx| async move {
while ctx.sleep(5.secs()).await {
ctx.http().get("https://example.com/").await?;
}
Ok(())
})
.restart(Restart::OnFailure)
.backoff(1.secs()..=30.secs());
w.rate_limit("example.com", 2.per_second());
w.schedule()
.call("cleanup", |_ctx| async move { Ok(()) })
.every(5.mins());
}
// bootstrap/app.rs: `app.agents(register)` with `use smeltery_watchfire::AgentsExt as _;`What it has:
- Supervision: restart policies (
Never,OnFailure,Always), exponential backoff with full jitter and a cap, a restart limit per time window, heartbeat stall detection with restart, panic capture, pause / resume, start / stop / restart, agents added and removed at runtime, pools (name#0..n), group and global concurrency limits, supervision trees (ctx.spawn_child), and exactly one recorded outcome per run, including on shutdown. AgentCtx: cancellation, heartbeat,sleep/interval, a polite HTTP client (timeouts, retries honouringRetry-After, per-host token buckets, a response size limit, a redirect policy that keeps credentials on their origin; reqwest with rustls and ring behind aTransporttrait), checkpoints, events (emit/on_event), counters saved in run records, logs kept in a ring buffer, the app and its database.- Jobs:
Jobstructs dispatched withjob.dispatch(&app), a database queue with atomic reservation, a Redis queue (featureredis: Lua scripts, atomic across processes) or an in-memory one, retries with backoff, dead letters (moved in and out in one step), stale reservation release. - Scheduler:
every,hourly,daily,daily_at,weekly, 5-fieldcron(UTC), overlapSkip/Queue/Allow; targets are jobs, closures and agents. - Persistence: the
watchfire_*tables over the app’s database (migrations::up/down), or memory. - App integration:
serveruns agents next to the web server (serve --no-agentsorWATCHFIRE_IN_SERVE=false: the web only),workruns them alone, both within the app’s shutdown budget; theagents:runs,schedule:listandschedule:runcommands. - Several processes: over a shared cache store’s atomic locks, each agent runs in one process at a time (a
renewed lease with a hard cut-off,
standbyelsewhere, takeover when the holder stops or dies) and each scheduled tick runs once; run history records each run’s process. The dashboard of any process shows the agents of the others from the shared tables and sends them commands throughwatchfire_commands. - Web: a JSON API and an SSE stream under
/_watchfire/api(bearer token; local development without it), a dashboard at/_watchfirefor signed-in users the app’sdashboard_gateadmits (a Mold page compiled into the crate with its own embedded stylesheet, live through Sparks, CSRF-protected command forms that ask before a stop or restart), and console commands (agents:list,agents:start|stop|pause|resume|restart,agents:logs) calling the running app, andagents:token. - Alerts:
on_alerthooks and a JSON webhook for failed agents, stalled runs and dead-lettered jobs. - LLM helpers (feature
llm): aProvidertrait,FakeProvider, theAnthropicMessages API adapter and a tool-callingAgentloop with typed tools, a turn limit, token / cost budgets and retries. - Testing:
testing::Harnesson paused time,testing::JobHarness(alone or inside a given app),http::FakeTransport.
§Licence
Dual-licensed under MIT or Apache-2.0, at your option.
Re-exports§
pub use http::Http;
Modules§
- http
- The polite HTTP client agents use:
Http. - llm
llm - LLM helpers (feature
llm): aProvidertrait with a scriptedFakeProviderand theAnthropicMessages API adapter, andAgent, a tool-calling loop with typed tools, a turn limit, a token / costBudgetand retries. - mail
mail - Mail with Watchfire (the
mailfeature): queued mail (QueueMail, theSendMailjob) and alert mail (WATCHFIRE_ALERT_MAIL). - migrations
- The framework tables Watchfire keeps in the app’s database.
- prelude
- Everything an agents or jobs file needs:
use smeltery::watchfire::prelude::*;. - testing
- Test helpers:
Harnessruns agents under the real supervisor on paused Tokio time with an in-memory store and a fake HTTP transport;JobHarnessruns a job’shandle. - web
- The JSON API, the SSE stream and the dashboard under
/_watchfire, and who may use them.
Structs§
- Agent
Builder - Settings for what
Watchfire::run/every/on_event/agent/poolregistered (every pool member). They override the agent’s ownconfig(). - Agent
Config - How the runtime supervises one agent. Every setter is also on the builder handle that
Watchfire::run/every/agent/poolreturn. - Agent
Ctx - What a run gets from the runtime: identity, cancellation, heartbeat, the app, HTTP, checkpoints, events, counters and logs. Cheap to clone; clones share everything.
- Agent
Error - The error an agent run, a job or a scheduled call returns.
- Agent
Log AgentCtx::log: one method per level.- Agent
Status - A snapshot of one agent.
- Agents
- The handle to the running agents: a service on the app once Watchfire runs
(
app.service::<Agents>()), and an extractor in handlers. Cheap to clone. - Alert
- One alert, as hooks receive it and the webhook posts it (JSON).
- Backoff
- Exponential backoff between automatic restarts, with full jitter.
- Cancellation
Token - A token which can be used to signal a cancellation request to one or more tasks.
- Counter
- A named counter of the current run, from
AgentCtx::counter. - Cron
- A 5-field cron expression (
minute hour day-of-month month day-of-week, UTC):*, lists (1,15), ranges (1-5), steps (*/10,0-30/5,5/15); day of week 0-7 (0 and 7 are Sunday). When both day fields are restricted, a day matching either runs (classic cron). - Dead
Letter - A job that failed for good.
- Event
- An event from
AgentCtx::emit, delivered toWatchfire::on_eventagents. - FnAgent
- An
Agentmade from a closure called once per run: whatWatchfire::runregisters, and whatagent_fnbuilds (forspawn_childand the testHarness). - Group
Builder Watchfire::group.- JobCtx
- What a job’s
handlereceives: the worker’sAgentCtx(throughDeref:log(),http(),db(),app(),counter()…) plus the job’s id and attempt. - LogLine
- A log line kept in an agent’s ring buffer (the last 200, see
Agents::logs). - Permit
- A concurrency permit from
AgentCtx::acquire; released on drop. - Queue
- The app’s queue: the driver, a wake-up for local workers and the clock. A service on the
app once Watchfire is set up (
.agents(...)). - Queue
Stats - Queue counts, for the API and dashboards.
- Rate
- A request rate for
Watchfire::rate_limit:countrequests perper, as a token bucket holding at mostcounttokens. - RunRecord
- One run of an agent (or one job / scheduled call handled by it), from start to end.
- Schedule
Builder Watchfire::schedule(): pick what runs.- Schedule
Info - The schedule as
schedule:listand the API show it. - Scheduled
Task - A scheduled task: pick when it runs and what happens on overlap.
- Store
Error - A failed store or queue operation.
- Ticker
- A cancellation-aware interval that heartbeats on every tick.
- Watchfire
- Everything an app runs in the background: agents, jobs, the schedule, groups and rate
limits.
app/agents/mod.rs’sregisterfunction fills it;AppBuilder::agents(fromAgentsExt) runs it. - Watchfire
Settings - Watchfire’s settings, from
.env/ the environment.
Enums§
- Agent
State - The lifecycle state of an agent.
- Alert
Kind - What happened.
- Error
- Errors from the Watchfire runtime: registration, commands and the queue.
- Health
- Heartbeat health of a running agent.
- Overlap
- What to do when a task is due while its previous run still runs.
- Restart
- When an agent whose run ended on its own is started again.
- RunOutcome
- How a run ended. Every run gets exactly one, including on shutdown.
Constants§
- MAX_
NAME_ LEN - Longest accepted agent, group or schedule name.
Traits§
- Agent
- A long-lived, supervised task: a poller, a scraper, a bot, a monitor.
- Agents
Ext AppBuilder::agents: run Watchfire in the app.- Duration
Ext 30.secs(),5.mins(),2.hours(): durations from integers.- Job
- A background job: a serializable value whose
handleruns on a queue worker. - RateExt
2.per_second(),30.per_minute():Rates from integers.
Functions§
- agent_
fn - An
Agentnamednamewhose every run callsf. - format_
utc YYYY-MM-DD HH:MM:SS(UTC) for Unix milliseconds.