Expand description
Typed message processing with explicit delivery and shutdown semantics.
Application code is a typed Input → Output Handler. The framework owns everything
around it: receiving from a Source, decoding, retries, concurrency, ordering,
publishing to a Sink, acknowledgement, dead letters, and graceful shutdown.
App
├── Subscription: Kafka source → handler → Kafka sink
├── Subscription: Pulsar source → handler → HTTP sink
└── Subscription: SQS source → handler → RabbitMQ sinkEach Subscription connects one source, one handler, and one sink. An App runs any
number of subscriptions side by side and shuts them down together.
§Quick start
Summarize each article on a Kafka topic with an LLM and publish the summaries to another topic. The handler sees only the decoded value; a transient API failure is retried, and each offset is committed after its summary is published.
use beavers::{
App, Classify, Utf8,
adapters::kafka::Kafka,
};
async fn summarize(article: String) -> beavers::Result<String> {
let summary = call_llm(&format!("Summarize: {article}")).await.retry()?;
Ok(summary)
}
#[tokio::main]
async fn main() -> Result<(), beavers::Error> {
let kafka = Kafka::new("localhost:9092");
let articles = kafka.source::<Utf8>("summarizer", ["articles"]);
let summaries = kafka.sink::<Utf8>("summaries");
App::new()
.subscribe("summarize", articles, summaries, summarize)
.run()
.await
}§Concepts
- A
Sourcereceives deliveries; each delivery (SourceMessage) owns its acknowledgement and is acknowledged only after its outputs are published. - A
Handlerturns a decoded input into an output, or into zero or many outputs withEmit. Its errors are classified withHandlerErrorandClassify. - A
Sinkprepares an output once and publishes it, with retries. - A
Subscriptionconnects one source, handler, and sink withMiddleware, a dead-letter sink, anErrorPolicy, and aSubscriptionConfig. - An
Appruns subscriptions together and shuts them down on SIGINT or SIGTERM. DecoderandEncoderimplementations such asJsonconvert payloads.
Delivery is at least once. Duplicates are possible after a failure or a broker
rebalance; Kafka-to-Kafka subscriptions can use TransactionalSink instead.
§Cargo features
| Feature | Enables |
|---|---|
kafka | Kafka source and sink in adapters::kafka |
pulsar | Apache Pulsar source and sink in adapters::pulsar |
rabbitmq | RabbitMQ source and sink in adapters::rabbitmq |
sqs | Amazon SQS source and sink in adapters::sqs |
pubsub | Google Cloud Pub/Sub source and sink in adapters::pubsub |
servicebus | Azure Service Bus source and sink in adapters::servicebus |
http | HTTP source and sink in adapters::http |
avro | The Avro codec |
protobuf | The Protobuf codec |
msgpack | The MessagePack codec |
cbor | The Cbor codec |
base64 | The Base64 codec wrapper |
gzip | The Gzip compression wrapper |
zstd | The Zstd compression wrapper |
opentelemetry | Trace-context propagation with TraceContext |
health | /livez and /readyz endpoints in health |
testing | Fabricated broker records for handler tests in testing |
No feature is enabled by default.
§Guides
Re-exports§
pub use adapters::ChannelOutput;pub use adapters::ChannelRaw;pub use adapters::ChannelReceiver;pub use adapters::ChannelSender;pub use adapters::ChannelSink;pub use adapters::ChannelSource;pub use adapters::InMemorySink;pub use adapters::IterSource;pub use adapters::StdinSource;pub use adapters::StdoutSink;pub use adapters::channel;pub use app::App;pub use blocking::Blocking;pub use blocking::BlockingPool;pub use blocking::blocking;pub use codec::Base64;base64pub use codec::Cbor;cborpub use codec::Gzip;gzippub use codec::MessagePack;msgpackpub use codec::Protobuf;protobufpub use codec::Zstd;zstdpub use codec::Avro;avropub use codec::Codec;pub use codec::Decoder;pub use codec::Encoder;pub use codec::Json;pub use codec::RawBytes;pub use codec::Utf8;pub use dead_letter::DEAD_LETTER_HEADER_PREFIX;pub use dead_letter::DeadLetter;pub use dead_letter::DeadLetterDetails;pub use error::BoxError;pub use error::Error;pub use error_policy::ErrorPolicy;pub use error_policy::FailureAction;pub use error_policy::FailureKind;pub use forward::SamePlatform;pub use forward::ValueRecord;pub use handler::Classify;pub use handler::Emit;pub use handler::Handler;pub use handler::HandlerError;pub use handler::Result;pub use health::Health;pub use message::Delivery;pub use message::OrderingKey;pub use message::SourceMessage;pub use middleware::Flow;pub use middleware::MapMetadata;pub use middleware::Middleware;pub use propagation::PropagationCarrier;pub use retry::Jitter;pub use retry::RetryPolicy;pub use shutdown::CancellationToken;pub use sink::Completion;pub use sink::PublishRejected;pub use sink::Sink;pub use source::Receive;pub use source::ReceiveError;pub use source::Source;pub use source::SourceItem;pub use source::SourceRaw;pub use subscription::ByRecord;pub use subscription::ByValue;pub use subscription::IntoHandler;pub use subscription::ProcessingOrder;pub use subscription::Subscription;pub use subscription::SubscriptionConfig;pub use subscription::TransactionBatch;pub use telemetry::TraceContext;opentelemetrypub use tombstone::PropagateTombstones;pub use tombstone::TombstonePublish;pub use tombstone::TombstoneRecord;pub use tombstone::Tombstones;pub use transaction::TransactionEntry;pub use transaction::TransactionalSink;
Modules§
- adapters
- Local transports and optional broker adapters.
- app
- Application registration, subscription supervision, and shutdown coordination.
- blocking
- Synchronous handlers executed on a dedicated, bounded worker pool.
- codec
- Serialization contracts and built-in codecs.
- dead_
letter - Envelope published to a dead-letter sink, and the failure details broker adapters forward with the original payload.
- error
- Errors the crate reports, and the boxed error accepted from application code.
- error_
policy - Per-failure outcomes for deliveries that cannot complete normally.
- forward
- Value-only handlers for sources and sinks that share a platform.
- handler
- Handler results, error classification, and output cardinality.
- health
- Liveness and readiness of an application’s subscriptions, readable through
Healthand, with thehealthfeature, served over HTTP byHealthServer. - message
- Received message ownership, decoding, ordering scope, and acknowledgement.
- middleware
- Typed subscription middleware around the handler.
- propagation
- Text-map fields that carry trace context, such as W3C
traceparent, across brokers. - retry
- Bounded exponential-backoff policy shared by independent retry domains.
- shutdown
- Cooperative cancellation and process termination signals.
- sink
- Preparing output is separate from transport publication and its retries.
- source
- Receive lifecycle and source resource ownership.
- subscription
- Subscription construction, configuration, and execution.
- telemetry
opentelemetry - OpenTelemetry trace-context propagation between deliveries and outputs.
- testing
testing - Utilities for testing handlers, middleware, and error policies without a
broker (
testingfeature). - tombstone
- Platform-neutral handling of tombstones: received records whose value is null, which compacted topics and change-data-capture streams use to delete a key.
- transaction
- Atomic publication and acknowledgement for compatible source and sink pairs.
Traits§
- Avro
Schema avro - Trait for types that serve as an Avro data model.