Crate ruststream_pulsar

Expand description

Apache Pulsar broker for the RustStream messaging framework, backed by the pulsar client maintained by StreamNative.

A Pulsar topic is a retained log, and a subscription is a durable cursor the server keeps on it. A subscription here is one descriptor, its type picks how its consumers share it, and a handler moves it over the log with a seek. Building the crate needs protoc on the path.

§A service

A handler is an async fn over the decoded payload, mounted on a broker inside the application object, and the attribute writes the main:

use std::time::Duration;

use ruststream_pulsar::prelude::*;
use serde::Deserialize;

#[derive(Deserialize)]
struct Order {
    id: u64,
}

#[subscriber(
    PulsarSubscription::new("orders", "workers")
        .subscription_type(SubscriptionType::Shared)
        .ack_timeout(Duration::from_secs(30))
)]
async fn handle(order: &Order) -> HandlerOutcome {
    tracing::info!(order.id, "accepted");
    HandlerOutcome::ack()
}

#[ruststream::app]
fn app() -> impl App {
    RustStream::new(AppInfo::new("orders", "0.1.0")).with_broker(
        PulsarBroker::new("pulsar://localhost:6650"),
        |b| {
            b.include(handle)
                .max_attempts(nonzero!(5))
                .dead_letter("orders-dlq");
        },
    )
}

cargo run -- run starts it. PulsarBroker::new only records the service URL; the runtime connects the broker at startup and shuts it down when the service stops.

§Where things are

The guides take a Pulsar service from installation to production, in this order: getting started, topics and subscriptions, subscription types and ordering, acknowledgement, retries and dead letters, replay and seeking, publishing and replies, testing and connecting to a production cluster. The architecture pages show how the crate sits between the pulsar client and the runtime, and the benchmarks what it costs.

Each topic below links to its component page on the site, which shows it in a service; the item’s own documentation keeps the detail.

§Broker

Pulsar broker: PulsarBroker, connected into ConnectedPulsarBroker.
Startup: the dial of the service URL and a consumer per subscription.
Shutdown: the producers closed, and PulsarError::NotConnected for a handle used afterwards.
Security: PulsarBroker::token and the pulsar+ssl:// listener.
OperationRetries: OperationRetries, how long the client retries an operation a server has not accepted.
max_message_size: PulsarBroker::max_message_size, the largest payload a publish may carry.
Prelude: prelude, the one glob a Pulsar service imports.
Errors: PulsarError.

§Subscriptions

Subscriptions: a bare topic name, read under the broker’s default_subscription.
PulsarSubscription: PulsarSubscription::new, one durable subscription on one topic.
topics: PulsarSubscription::topics, a fixed list of topics.
pattern: PulsarSubscription::pattern, every topic whose full name a regular expression matches.
SubscriptionType: SubscriptionType, how the consumers of one subscription share it.
PulsarTopic: PulsarTopic, a topic name checked on construction.
Batches: a slice handler and PulsarSubscription::batch_wait.

§Settlement

Acknowledgement: what ack, retry and drop do on a PulsarMessage.
ack_timeout: PulsarSubscription::ack_timeout, the redelivery of an unsettled message.
retry_after: a negative acknowledgement held back for a delay.
Dead letters: max_attempts with dead_letter, the consumer’s own dead-letter policy.

§Context

PulsarContext: PulsarContext and its keys Position and SeekHandle.
PulsarBatchContext: PulsarBatchContext, a batch handler’s context.

§Positions

PulsarPosition: PulsarPosition, a place in a topic’s retained log.
start_at: a subscription opened at a position on every startup.
Seek: SeekHandle and PulsarSeeker, moving a running subscription.

§Publishing

Publish: PulsarPublish, the publish policy, and its live PulsarPublisher.
partition_key: PulsarPublishSteps::partition_key and PulsarPublishOptions key one publish.
Keys: PARTITION_KEY_HEADER becomes the message key.
Headers: one message property per header.

§Tooling

AsyncAPI: the Pulsar bindings of the generated document (feature asyncapi): server, channels and operations.
Testing: the in-process Pulsar TestApp connects the broker to (feature testing), and Live runs against a running Pulsar.

§Cargo features

Both features are additive and off by default. testing adds the in-process mode TestApp::start connects the broker through, and asyncapi contributes the Pulsar bindings to the generated document. The codecs come from ruststream’s own features.

Modules§

prelude: The imports a service on Pulsar writes every time, in one glob.

Structs§

ConnectedPulsarBroker: The typed witness that connect succeeded: holds the live client directly.
OperationRetries: How long the client keeps asking for an operation a server has not accepted yet.
Position: The key reading this delivery’s PulsarPosition out of PulsarContext.
PulsarBatchContext: The subscription-scoped batch context of a Pulsar subscription: its seeker, shared by every delivery of the batch.
PulsarBroker: An Apache Pulsar broker for the RustStream messaging framework.
PulsarContext: The per-delivery context of a Pulsar subscription: this message’s position in the topic’s retained log, and the subscription’s own seeker.
PulsarMessage: A message delivered by a PulsarSubscriber.
PulsarPublish: The publish policy for PulsarPublisher: pure declaration, constructible anywhere, paired with the connected broker by the runtime after connect.
PulsarPublishOptions: What one Pulsar publish differs from the next in.
PulsarPublisher: Publishes messages to Pulsar topics, one producer per topic, created lazily and shared through the broker core (so shutdown can close them).
PulsarSeeker: Repositions a subscription while its stream runs; minted by Seekable::seeker.
PulsarSubscriber: A subscription to one or more Pulsar topics; yields PulsarMessages, one at a time or in batches.
PulsarSubscription: A subscription descriptor for one Pulsar subscription over one or more topics.
PulsarTopic: A validated Pulsar topic name.
SeekHandle: The key reading the subscription’s PulsarSeeker out of PulsarContext (and out of PulsarBatchContext on the batch path): the reposition handle every delivery of the subscription shares.

Enums§

PulsarError: Errors returned by the Apache Pulsar broker.
PulsarPosition: A position in a topic’s retained log, accepted by Seeker::seek.
SubscriptionType: How competing consumers on one subscription share its messages.

Constants§

PARTITION_KEY_HEADER: Header carrying the partition key, mapped onto the message’s partition_key (which KeyShared subscriptions order by).

Traits§

PulsarPublishSteps: Pulsar’s per-message steps on the framework’s publish builder.