Skip to main content

Crate smeltery_watchfire

Crate smeltery_watchfire 

Source
Expand description

Watchfire, the agent runtime of the Smeltery framework: supervised long-running agents, background jobs on a queue, and a scheduler, on one Tokio runtime. Apps use it as smeltery::watchfire.

§smeltery-watchfire

Watchfire is the agent runtime of the Smeltery framework: supervised long-running agents, background jobs on a queue, and a scheduler, on one Tokio runtime. Apps use it through the facade as smeltery::watchfire; the full guide is the “Watchfire” section of the Smeltery README.

use smeltery_watchfire::prelude::*;

pub fn register(w: &mut Watchfire) {
    w.every(30.secs(), "heartbeat", |ctx| async move {
        ctx.log().info("tick");
        Ok(())
    });
    w.run("scraper", |ctx| async move {
        while ctx.sleep(5.secs()).await {
            ctx.http().get("https://example.com/").await?;
        }
        Ok(())
    })
    .restart(Restart::OnFailure)
    .backoff(1.secs()..=30.secs());
    w.rate_limit("example.com", 2.per_second());
    w.schedule()
        .call("cleanup", |_ctx| async move { Ok(()) })
        .every(5.mins());
}

// bootstrap/app.rs: `app.agents(register)` with `use smeltery_watchfire::AgentsExt as _;`

What it has:

  • Supervision: restart policies (Never, OnFailure, Always), exponential backoff with full jitter and a cap, a restart limit per time window, heartbeat stall detection with restart, panic capture, pause / resume, start / stop / restart, agents added and removed at runtime, pools (name#0..n), group and global concurrency limits, supervision trees (ctx.spawn_child), and exactly one recorded outcome per run, including on shutdown.
  • AgentCtx: cancellation, heartbeat, sleep / interval, a polite HTTP client (timeouts, retries honouring Retry-After, per-host token buckets, a response size limit, a redirect policy that keeps credentials on their origin; reqwest with rustls and ring behind a Transport trait), checkpoints, events (emit / on_event), counters saved in run records, logs kept in a ring buffer, the app and its database.
  • Jobs: Job structs dispatched with job.dispatch(&app), a database queue with atomic reservation, a Redis queue (feature redis: Lua scripts, atomic across processes) or an in-memory one, retries with backoff, dead letters (moved in and out in one step), stale reservation release.
  • Scheduler: every, hourly, daily, daily_at, weekly, 5-field cron (UTC), overlap Skip / Queue / Allow; targets are jobs, closures and agents.
  • Persistence: the watchfire_* tables over the app’s database (migrations::up / down), or memory.
  • App integration: serve runs agents next to the web server (serve --no-agents or WATCHFIRE_IN_SERVE=false: the web only), work runs them alone, both within the app’s shutdown budget; the agents:runs, schedule:list and schedule:run commands.
  • Several processes: over a shared cache store’s atomic locks, each agent runs in one process at a time (a renewed lease with a hard cut-off, standby elsewhere, takeover when the holder stops or dies) and each scheduled tick runs once; run history records each run’s process. The dashboard of any process shows the agents of the others from the shared tables and sends them commands through watchfire_commands.
  • Web: a JSON API and an SSE stream under /_watchfire/api (bearer token; local development without it), a dashboard at /_watchfire for signed-in users the app’s dashboard_gate admits (a Mold page compiled into the crate with its own embedded stylesheet, live through Sparks, CSRF-protected command forms that ask before a stop or restart), and console commands (agents:list, agents:start|stop|pause|resume|restart, agents:logs) calling the running app, and agents:token.
  • Alerts: on_alert hooks and a JSON webhook for failed agents, stalled runs and dead-lettered jobs.
  • LLM helpers (feature llm): a Provider trait, FakeProvider, the Anthropic Messages API adapter and a tool-calling Agent loop with typed tools, a turn limit, token / cost budgets and retries.
  • Testing: testing::Harness on paused time, testing::JobHarness (alone or inside a given app), http::FakeTransport.

§Licence

Dual-licensed under MIT or Apache-2.0, at your option.

Re-exports§

pub use http::Http;

Modules§

http
The polite HTTP client agents use: Http.
llmllm
LLM helpers (feature llm): a Provider trait with a scripted FakeProvider and the Anthropic Messages API adapter, and Agent, a tool-calling loop with typed tools, a turn limit, a token / cost Budget and retries.
mailmail
Mail with Watchfire (the mail feature): queued mail (QueueMail, the SendMail job) and alert mail (WATCHFIRE_ALERT_MAIL).
migrations
The framework tables Watchfire keeps in the app’s database.
prelude
Everything an agents or jobs file needs: use smeltery::watchfire::prelude::*;.
testing
Test helpers: Harness runs agents under the real supervisor on paused Tokio time with an in-memory store and a fake HTTP transport; JobHarness runs a job’s handle.
web
The JSON API, the SSE stream and the dashboard under /_watchfire, and who may use them.

Structs§

AgentBuilder
Settings for what Watchfire::run / every / on_event / agent / pool registered (every pool member). They override the agent’s own config().
AgentConfig
How the runtime supervises one agent. Every setter is also on the builder handle that Watchfire::run / every / agent / pool return.
AgentCtx
What a run gets from the runtime: identity, cancellation, heartbeat, the app, HTTP, checkpoints, events, counters and logs. Cheap to clone; clones share everything.
AgentError
The error an agent run, a job or a scheduled call returns.
AgentLog
AgentCtx::log: one method per level.
AgentStatus
A snapshot of one agent.
Agents
The handle to the running agents: a service on the app once Watchfire runs (app.service::<Agents>()), and an extractor in handlers. Cheap to clone.
Alert
One alert, as hooks receive it and the webhook posts it (JSON).
Backoff
Exponential backoff between automatic restarts, with full jitter.
CancellationToken
A token which can be used to signal a cancellation request to one or more tasks.
Counter
A named counter of the current run, from AgentCtx::counter.
Cron
A 5-field cron expression (minute hour day-of-month month day-of-week, UTC): *, lists (1,15), ranges (1-5), steps (*/10, 0-30/5, 5/15); day of week 0-7 (0 and 7 are Sunday). When both day fields are restricted, a day matching either runs (classic cron).
DeadLetter
A job that failed for good.
Event
An event from AgentCtx::emit, delivered to Watchfire::on_event agents.
FnAgent
An Agent made from a closure called once per run: what Watchfire::run registers, and what agent_fn builds (for spawn_child and the test Harness).
GroupBuilder
Watchfire::group.
JobCtx
What a job’s handle receives: the worker’s AgentCtx (through Deref: log(), http(), db(), app(), counter() …) plus the job’s id and attempt.
LogLine
A log line kept in an agent’s ring buffer (the last 200, see Agents::logs).
Permit
A concurrency permit from AgentCtx::acquire; released on drop.
Queue
The app’s queue: the driver, a wake-up for local workers and the clock. A service on the app once Watchfire is set up (.agents(...)).
QueueStats
Queue counts, for the API and dashboards.
Rate
A request rate for Watchfire::rate_limit: count requests per per, as a token bucket holding at most count tokens.
RunRecord
One run of an agent (or one job / scheduled call handled by it), from start to end.
ScheduleBuilder
Watchfire::schedule(): pick what runs.
ScheduleInfo
The schedule as schedule:list and the API show it.
ScheduledTask
A scheduled task: pick when it runs and what happens on overlap.
StoreError
A failed store or queue operation.
Ticker
A cancellation-aware interval that heartbeats on every tick.
Watchfire
Everything an app runs in the background: agents, jobs, the schedule, groups and rate limits. app/agents/mod.rs’s register function fills it; AppBuilder::agents (from AgentsExt) runs it.
WatchfireSettings
Watchfire’s settings, from .env / the environment.

Enums§

AgentState
The lifecycle state of an agent.
AlertKind
What happened.
Error
Errors from the Watchfire runtime: registration, commands and the queue.
Health
Heartbeat health of a running agent.
Overlap
What to do when a task is due while its previous run still runs.
Restart
When an agent whose run ended on its own is started again.
RunOutcome
How a run ended. Every run gets exactly one, including on shutdown.

Constants§

MAX_NAME_LEN
Longest accepted agent, group or schedule name.

Traits§

Agent
A long-lived, supervised task: a poller, a scraper, a bot, a monitor.
AgentsExt
AppBuilder::agents: run Watchfire in the app.
DurationExt
30.secs(), 5.mins(), 2.hours(): durations from integers.
Job
A background job: a serializable value whose handle runs on a queue worker.
RateExt
2.per_second(), 30.per_minute(): Rates from integers.

Functions§

agent_fn
An Agent named name whose every run calls f.
format_utc
YYYY-MM-DD HH:MM:SS (UTC) for Unix milliseconds.

Type Aliases§

BoxError
A boxed, thread-safe error.
JobId
A queued job’s id.