Skip to main content

Crate beavers

Crate beavers 

Source
Expand description

Typed message processing with explicit delivery and shutdown semantics.

Application code is a typed Input → Output Handler. The framework owns everything around it: receiving from a Source, decoding, retries, concurrency, ordering, publishing to a Sink, acknowledgement, dead letters, and graceful shutdown.

App
├── Subscription: Kafka source  → handler → Kafka sink
├── Subscription: Pulsar source → handler → HTTP sink
└── Subscription: SQS source    → handler → RabbitMQ sink

Each Subscription connects one source, one handler, and one sink. An App runs any number of subscriptions side by side and shuts them down together.

§Quick start

Summarize each article on a Kafka topic with an LLM and publish the summaries to another topic. The handler sees only the decoded value; a transient API failure is retried, and each offset is committed after its summary is published.

use beavers::{
    App, Classify, Utf8,
    adapters::kafka::Kafka,
};

async fn summarize(article: String) -> beavers::Result<String> {
    let summary = call_llm(&format!("Summarize: {article}")).await.retry()?;
    Ok(summary)
}

#[tokio::main]
async fn main() -> Result<(), beavers::Error> {
    let kafka = Kafka::new("localhost:9092");
    let articles = kafka.source::<Utf8>("summarizer", ["articles"]);
    let summaries = kafka.sink::<Utf8>("summaries");

    App::new()
        .subscribe("summarize", articles, summaries, summarize)
        .run()
        .await
}

§Concepts

  • A Source receives deliveries; each delivery (SourceMessage) owns its acknowledgement and is acknowledged only after its outputs are published.
  • A Handler turns a decoded input into an output, or into zero or many outputs with Emit. Its errors are classified with HandlerError and Classify.
  • A Sink prepares an output once and publishes it, with retries.
  • A Subscription connects one source, handler, and sink with Middleware, a dead-letter sink, an ErrorPolicy, and a SubscriptionConfig.
  • An App runs subscriptions together and shuts them down on SIGINT or SIGTERM.
  • Decoder and Encoder implementations such as Json convert payloads.

Delivery is at least once. Duplicates are possible after a failure or a broker rebalance; Kafka-to-Kafka subscriptions can use TransactionalSink instead.

§Cargo features

FeatureEnables
kafkaKafka source and sink in adapters::kafka
pulsarApache Pulsar source and sink in adapters::pulsar
rabbitmqRabbitMQ source and sink in adapters::rabbitmq
sqsAmazon SQS source and sink in adapters::sqs
pubsubGoogle Cloud Pub/Sub source and sink in adapters::pubsub
servicebusAzure Service Bus source and sink in adapters::servicebus
httpHTTP source and sink in adapters::http
avroThe Avro codec
protobufThe Protobuf codec
msgpackThe MessagePack codec
cborThe Cbor codec
base64The Base64 codec wrapper
gzipThe Gzip compression wrapper
zstdThe Zstd compression wrapper
opentelemetryTrace-context propagation with TraceContext
health/livez and /readyz endpoints in health
testingFabricated broker records for handler tests in testing

No feature is enabled by default.

§Guides

Re-exports§

pub use adapters::ChannelOutput;
pub use adapters::ChannelRaw;
pub use adapters::ChannelReceiver;
pub use adapters::ChannelSender;
pub use adapters::ChannelSink;
pub use adapters::ChannelSource;
pub use adapters::InMemorySink;
pub use adapters::IterSource;
pub use adapters::StdinSource;
pub use adapters::StdoutSink;
pub use adapters::channel;
pub use app::App;
pub use blocking::Blocking;
pub use blocking::BlockingPool;
pub use blocking::blocking;
pub use codec::Base64;base64
pub use codec::Cbor;cbor
pub use codec::Gzip;gzip
pub use codec::MessagePack;msgpack
pub use codec::Protobuf;protobuf
pub use codec::Zstd;zstd
pub use codec::Avro;avro
pub use codec::Codec;
pub use codec::Decoder;
pub use codec::Encoder;
pub use codec::Json;
pub use codec::RawBytes;
pub use codec::Utf8;
pub use dead_letter::DEAD_LETTER_HEADER_PREFIX;
pub use dead_letter::DeadLetter;
pub use dead_letter::DeadLetterDetails;
pub use error::BoxError;
pub use error::Error;
pub use error_policy::ErrorPolicy;
pub use error_policy::FailureAction;
pub use error_policy::FailureKind;
pub use forward::SamePlatform;
pub use forward::ValueRecord;
pub use handler::Classify;
pub use handler::Emit;
pub use handler::Handler;
pub use handler::HandlerError;
pub use handler::Result;
pub use health::Health;
pub use message::Delivery;
pub use message::OrderingKey;
pub use message::SourceMessage;
pub use middleware::Flow;
pub use middleware::MapMetadata;
pub use middleware::Middleware;
pub use propagation::PropagationCarrier;
pub use retry::Jitter;
pub use retry::RetryPolicy;
pub use shutdown::CancellationToken;
pub use sink::Completion;
pub use sink::PublishRejected;
pub use sink::Sink;
pub use source::Receive;
pub use source::ReceiveError;
pub use source::Source;
pub use source::SourceItem;
pub use source::SourceRaw;
pub use subscription::ByRecord;
pub use subscription::ByValue;
pub use subscription::IntoHandler;
pub use subscription::ProcessingOrder;
pub use subscription::Subscription;
pub use subscription::SubscriptionConfig;
pub use subscription::TransactionBatch;
pub use telemetry::TraceContext;opentelemetry
pub use tombstone::PropagateTombstones;
pub use tombstone::TombstonePublish;
pub use tombstone::TombstoneRecord;
pub use tombstone::Tombstones;
pub use transaction::TransactionEntry;
pub use transaction::TransactionalSink;

Modules§

adapters
Local transports and optional broker adapters.
app
Application registration, subscription supervision, and shutdown coordination.
blocking
Synchronous handlers executed on a dedicated, bounded worker pool.
codec
Serialization contracts and built-in codecs.
dead_letter
Envelope published to a dead-letter sink, and the failure details broker adapters forward with the original payload.
error
Errors the crate reports, and the boxed error accepted from application code.
error_policy
Per-failure outcomes for deliveries that cannot complete normally.
forward
Value-only handlers for sources and sinks that share a platform.
handler
Handler results, error classification, and output cardinality.
health
Liveness and readiness of an application’s subscriptions, readable through Health and, with the health feature, served over HTTP by HealthServer.
message
Received message ownership, decoding, ordering scope, and acknowledgement.
middleware
Typed subscription middleware around the handler.
propagation
Text-map fields that carry trace context, such as W3C traceparent, across brokers.
retry
Bounded exponential-backoff policy shared by independent retry domains.
shutdown
Cooperative cancellation and process termination signals.
sink
Preparing output is separate from transport publication and its retries.
source
Receive lifecycle and source resource ownership.
subscription
Subscription construction, configuration, and execution.
telemetryopentelemetry
OpenTelemetry trace-context propagation between deliveries and outputs.
testingtesting
Utilities for testing handlers, middleware, and error policies without a broker (testing feature).
tombstone
Platform-neutral handling of tombstones: received records whose value is null, which compacted topics and change-data-capture streams use to delete a key.
transaction
Atomic publication and acknowledgement for compatible source and sink pairs.

Traits§

AvroSchemaavro
Trait for types that serve as an Avro data model.