Pilgrimage is a Rust implementation of a distributed messaging system inspired by Apache Kafka. It records messages to local files and supports At-least-once and Exactly-once delivery semantics.
- Installation
- Security
- Features
- Basic Usage
- CLI Features
- Web Console API
- Version increment on release
Installation
To use Pilgrimage, add the following to your Cargo.toml:
[]
= "0.14.0"
Security
When using Pilgramage as a Crate, client authentication is implemented, but at present, authentication is not implemented for message sending and receiving from the CLI and web client. You can find a sample of authentication with Crate examples/auth-example.rs, examples/auth-send-recv.rs.
Features
- Topic-based pub/sub model
- Scalability through partitioning
- Persistent messages (log file based)
- Leader/Follower Replication
- Fault Detection and Automatic Recovery
- Delivery guaranteed by acknowledgement (ACK)
- Fully implemented leader selection mechanism
- Partition Replication
- Persistent messages
- Schema Registry for managing message schemas and ensuring compatibility
- Automatic Scaling
- Broker Clustering
- Message processing in parallel
- Authentication and Authorization Mechanisms
- Data Encryption
- CLI based console
- WEB based console
- Support AMQP
Basic Usage
use ;
use SchemaRegistry;
use Utc;
use Uuid;
use ;
use ;
async
Dependency
- Rust 1.51.0 or later
Functionality Implemented
- Message Queue: Efficient message queue implementation using
MutexandVecDeque. - Broker: Core broker functionality including message handling, node management, and leader election.
- Consumer Groups: Support for consumer groups to allow multiple consumers to read from the same topic.
- Leader Election: Mechanism for electing a leader among brokers to manage partitions and replication.
- Storage: Persistent storage of messages using local files.
- Replication: Replication of messages across multiple brokers for fault tolerance.
- Schema Registry: Management of message schemas to ensure compatibility between producers and consumers.
- Benchmarking: Comprehensive benchmarking tests to measure performance of various components.
- Automatic Scaling: Automatically scale the number of instances based on load.
- Log Compressions: Compress and optimize logs.
Examples
To execute a basic example, use the following command:
Benchmarks
Pilgrimage includes a comprehensive suite of benchmarks to measure performance in a variety of scenarios.
execution method
Benchmark Category
-
Message Sending - Transmission performance with different message sizes
- Small messages (~12 bytes): ~6.0 µs
- Medium messages (1KB): ~16.2 µs
- Large messages (10KB): ~19.6 µs
-
Message Receiving - Message reception performance
- Average receive time: ~82.7 µs
-
Topic Operations - Topic Management Operations
- Topic creation: ~1.6 µs
- Topic listing: ~652 ms (warning: slow operation)
-
Partition Operations - Transmission performance by partition
- 1 partition: ~7.2 µs
- 2 partitions: ~13.9 µs
- 4 partitions: ~28.5 µs
- 8 partitions: ~54.7 µs
-
Concurrent Operations - Parallel transmission and reception performance
- Send + Receive: ~5.5 ms
-
Throughput Testing - Batch Processing Performance
- 10 messages: ~69.0 µs
- 100 messages: ~693 µs
- 1000 messages: ~6.7 ms
Checking Reports
After the benchmark is run, a detailed HTML report is generated in target/criterion/report/index.html.
If the allocated memory is small, it may fail.
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
To run the benchmark on your local machine, use the command:
CLI Features
Pilgrimage offers a comprehensive Command-Line Interface (CLI) to manage and interact with your message brokers efficiently. Below are the available commands along with their descriptions and usage examples.
start
Description: Starts the broker with the specified configurations.
Usage:
Options:
--id,-i(required): Sets the broker ID.--partitions,-p(required): Sets the number of partitions.--replication,-r(required): Sets the replication factor.--storage,-s(required): Sets the storage path.--test-mode: Runs the broker in test mode, which breaks out of the main loop quickly for testing purposes.
Example:
pilgrimage start --id broker1 --partitions 3 --replication 2 --storage /data/broker1 --test-mode
stop
Description: Stops the specified broker.
Usage
pilgrimage stop --id <BROKER_ID>
Options:
--id,-i(required): Sets the broker ID.
Example
pilgrimage stop --id broker1
send
Description:
Sends a message to a topic.
Usage
pilgrimage send --topic <TOPIC> --message <MESSAGE> [--schema <SCHEMA>] [--compatibility <COMPATIBILITY>]
Options:
--topic,-t(required): Specifies the topic name.--message,-m(required): Specifies the message to send.--schema,-s(optional): Specifies the path to a schema file. If not specified, an existing schema will be used.--compatibility,-c(optional): Specifies the schema compatibility level (BACKWARD, FORWARD, FULL, NONE).
Example
pilgrimage send --topic test_topic --message "Hello, World!"
consume
Description:
Consumes messages from a broker.
Usage
pilgrimage consume --id <BROKER_ID> [--topic <TOPIC>] [--partition <PARTITION>] [--group <GROUP>]
Options:
--id,-i(required): Specifies the broker ID.--topic,-t(optional): Specifies the topic name.--partition,-p(optional): Specifies the partition number.--group,-g(optional): Specifies the consumer group ID.
Example:
pilgrimage consume --id broker1 --topic test_topic --partition 0
status
Description:
Checks the status of the specified broker.
Usage:
pilgrimage status --id <BROKER_ID>
Options:
--id,-i(required): Sets the broker ID.
Example:
pilgrimage status --id broker1
schema
Description:
Manages message schemas for topics.
Subcommands:
-
register - Register a new schema for a topic
Usage:
pilgrimage schema register --topic <TOPIC> --schema <SCHEMA_FILE> [--compatibility <COMPATIBILITY>]Options:
--topic,-t(required): Specifies the topic name to register the schema for.--schema,-s(required): Specifies the path to the schema file.--compatibility,-c(optional): Specifies the schema compatibility level (BACKWARD, FORWARD, FULL, NONE).
Example:
pilgrimage schema register --topic test_topic --schema ./schemas/test_schema.json --compatibility BACKWARD -
list - List all schemas for a topic
Usage:
pilgrimage schema list --topic <TOPIC>Options:
--topic,-t(required): Specifies the topic name.
Example:
pilgrimage schema list --topic test_topic
Additional Information
- Help Command:
To view all available commands and options, use the
helpcommand:
pilgrimage help
- Version Information: To check the current version of Pilgrimage, use:
pilgrimage --version
Running the CLI
To run the CLI application:
Examples:
# Start a broker
# Send a message to a topic
# Consume messages from a broker
Web Console API
Pilgrimage provides a REST API for managing brokers through HTTP requests. The server runs on http://localhost:8080 by default.
Available Endpoints
Start Broker
Starts a new broker instance.
Endpoint: POST /start
Request:
Example:
Stop Broker
Stops a running broker instance.
Endpoint: POST /stop
Request:
Example:
Send Message
Sends a message to the broker. If the specified topic does not exist, it will be created automatically.
Endpoint: POST /send
Request:
Example:
Consume Messages
Consumes messages from the broker. If the specified topic does not exist, it will be created automatically.
Endpoint: POST /consume
Request:
Example:
# Default topic (default_topic)
# Custom topic
Check Status
Checks the status of the broker.
Endpoint: POST /status
Request:
{
}
Example:
Running the Web Server
To start the web server:
The server will be available at http://localhost:8080.
Version increment on release
- The commit message is parsed and the version of either major, minor or patch is incremented.
- The version of Cargo.toml is updated.
- The updated Cargo.toml is committed and a new tag is created.
- The changes and tag are pushed to the remote repository.
The version is automatically incremented based on the commit message. Here, we treat feat as minor, fix as patch, and BREAKING CHANGE as major.
License
MIT