Module kafka::consumer

Kafka Consumer - A higher-level API for consuming a kafka topic.

A consumer for a single Kafka topic on behalf of a specified group providing help in offset management. The consumer can be optionally advised to consume only particular partitions of the underlying topic.

Example

use kafka::consumer::{Consumer, FetchOffset};

let mut consumer =
   Consumer::from_hosts(vec!("localhost:9092".to_owned()),
                             "my-group".to_owned(),
                             "my-topic".to_owned())
      .with_partitions(&[0, 1])
      .with_fallback_offset(FetchOffset::Earliest)
      .create()
      .unwrap();
loop {
  for ms in consumer.poll().unwrap().iter() {
    for m in ms.messages() {
      println!("{:?}", m);
    }
    consumer.consume_messageset(ms);
  }
  consumer.commit_consumed();
}

A .poll() will ask for the next available "chunk of data" for client code to process. The returned data are MessageSets - at most one for each partition of the consumed topic.

The consumer helps in keeping track of already consumed messages by maintaining a map of the consumed offsets. Messages can be told "consumed" either through consume_message or consume_messages methods. Once these consumed messages are committed to Kafka using commit_consumed, the consumer will start fetching messages from here after restart. Since committing is a certain overhead, it is up to the client to decide the frequency of the commits. The consumer will not commit any messages to Kafka automatically.

Reexports

pub use client::FetchOffset;

Structs

Builder	A Kafka Consumer builder easing the process of setting up various configuration settings.
Consumer	The Kafka Consumer
Message	A fetched message from a remote Kafka broker for a particular topic partition.
MessageSet	A set of messages succesfully retrieved from a specific topic partition.
MessageSets	Messages retrieved from kafka in one fetch request. This is a concatenation of blocks of messages successfully retrieved from the consumed topic partitions. Each such partitions is guaranteed to be present at most once in this structure.
MessageSetsIter	An iterator over the consumed topic partition message sets.

Constants

DEFAULT_RETRY_MAX_BYTES_LIMIT

The default value for Builder::with_retry_max_bytes_limit.