Crate fluvio

Expand description

Fluvio

The programmable data streaming platform

§What’s Fluvio?

Fluvio is a programmable data streaming platform written in Rust. With Fluvio you can create performant real time applications that scale.

Read more about Fluvio in the official website.

§Getting Started

Let’s write a very simple solution with Fluvio, in the following demostration we will create a topic using the Fluvio CLI and then we wisll produce some records on this topic. Finally these records will be consumed from the topic and printed to the stdout.

Install Fluvio CLI if you havent already
Create a new topic using the CLI

fluvio topic create "echo-test"

Create a new cargo project and install fluvio, futures and async-std

cargo add fluvio
cargo add futures
cargo add async-std --features attributes

Copy and paste the following snippet into your src/main.rs

use std::time::Duration;

use fluvio::{Offset, RecordKey};
use futures::StreamExt;

const TOPIC: &str = "echo-test";
const MAX_RECORDS: u8 = 10;

#[async_std::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let producer = fluvio::producer(TOPIC).await?;
    let consumer = fluvio::consumer(TOPIC, 0).await?;
    let mut consumed_records: u8 = 0;

    for i in 0..10 {
        producer.send(RecordKey::NULL, format!("Hello from Fluvio {}!", i)).await?;
        println!("[PRODUCER] sent record {}", i);
        async_std::task::sleep(Duration::from_secs(1)).await;
    }

    // Fluvio batches records by default, call flush() when done producing
    // to ensure all records are sent
    producer.flush().await?;

    let mut stream = consumer.stream(Offset::beginning()).await?;

    while let Some(Ok(record)) = stream.next().await {
        let value_str = record.get_value().as_utf8_lossy_string();

        println!("[CONSUMER] Got record: {}", value_str);
        consumed_records += 1;

        if consumed_records >= MAX_RECORDS {
            break;
        }
    }

    Ok(())
}

Run cargo run and expect the following output

[PRODUCER] sent record 0
[PRODUCER] sent record 1
[PRODUCER] sent record 2
[PRODUCER] sent record 3
[PRODUCER] sent record 4
[PRODUCER] sent record 5
[PRODUCER] sent record 6
[PRODUCER] sent record 7
[PRODUCER] sent record 8
[PRODUCER] sent record 9
[CONSUMER] Got record: Hello, Fluvio 0!
[CONSUMER] Got record: Hello, Fluvio 1!
[CONSUMER] Got record: Hello, Fluvio 2!
[CONSUMER] Got record: Hello, Fluvio 3!
[CONSUMER] Got record: Hello, Fluvio 4!
[CONSUMER] Got record: Hello, Fluvio 5!
[CONSUMER] Got record: Hello, Fluvio 6!
[CONSUMER] Got record: Hello, Fluvio 7!
[CONSUMER] Got record: Hello, Fluvio 8!
[CONSUMER] Got record: Hello, Fluvio 9!

Clean Up

fluvio topic delete echo-test
topic "echo-test" deleted

§Learn More

Read on tutorials to get the most from Fluvio and InfinyOn Cloud to scale your streaming solution.
You can use Fluvio to send or receive records from different sources using Connectors.
If you want to filter or transform records on the fly read more about SmartModules.

Re-exports§

pub use config::FluvioClusterConfig;
pub use config::FluvioConfig;
pub use consumer::PartitionConsumer;
pub use consumer::ConsumerConfig;
pub use consumer::MultiplePartitionConsumer;
pub use consumer::PartitionSelectionStrategy;

Modules§

config
consumer
dataplane
metadata: re-export metadata from sc-api
metrics
spu

Structs§

Fluvio: An interface for interacting with Fluvio streaming
FluvioAdmin: An interface for managing a Fluvio cluster
FutureRecordMetadata: Output of TopicProducer::send Used to wait the RecordMetadata of the record being sent. See FutureRecordMetadata::wait
Offset: Describes the location of an event stored in a Fluvio partition
PartitionerConfig
ProduceCompletionBatchEvent: An event that is triggered when a batch is full.
ProduceOutput: Struct returned by of TopicProduce::send call, it is used to gather the record metadata associated to each send call.
RecordKey: A key for determining which partition a record should be sent to.
RecordMetadata: Metadata of a record send to a topic
RetryPolicy: Defines parameters of retries in DeliverySemantic::AtLeastOnce delivery semantic.
SmartModuleExtraParams
SmartModuleInvocation: The request payload when using a Consumer SmartModule.
TopicProducer: An interface for producing events to a particular topic
TopicProducerConfig: Options used to adjust the behavior of the Producer. Create this struct with TopicProducerConfigBuilder.
TopicProducerConfigBuilder: Builder for TopicProducerConfig.

Enums§

Compression: The compression algorithm used to compress and decompress records in fluvio batches
DeliverySemantic: Defines guarantees that Producer must follow delivering records to SPU.
FluvioError: Possible errors that may arise when using Fluvio
Isolation
ProducerError
RetryStrategy: Strategy of delays distribution.
SmartModuleContextData
SmartModuleInvocationWasm
SmartModuleKind: Indicates the type of SmartModule as well as any special data required

Traits§

Partitioner: A trait for defining a partitioning strategy for key/value records.
ProducerCallback: A trait for the producer callback.

Functions§

consumerDeprecated: Creates a consumer that receives events from the given topic and partition
producer: Creates a producer that sends records to the named topic

Type Aliases§

PartitionId
SharedProducerCallback: A shared trait object for the producer callback.
TopicProducerPool: Pool of producers for a given topic. There is a producer per partition

Crate fluvioCopy item path