Module offset_manager 

Kafka offset management and checkpointing

This module provides comprehensive offset tracking, persistence, and recovery capabilities for Kafka consumers. It integrates with the state backend system to ensure exactly-once processing semantics and fault tolerance.

Features

• Per-partition offset tracking: Track offsets independently for each partition
• Watermark-based commits: Commit offsets only after watermark passes
• Checkpoint integration: Save offsets as part of state checkpoints
• Offset lag monitoring: Calculate and track consumer lag
• Reset strategies: Support for earliest, latest, and specific offset resets
• Automatic commit strategies: Periodic, watermark-based, or manual commits

Example

use processor::kafka::{OffsetManager, OffsetCommitStrategy, InMemoryOffsetStore};
use processor::state::MemoryStateBackend;
use std::sync::Arc;
use std::time::Duration;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let backend = Arc::new(MemoryStateBackend::new(None));
    let store = Arc::new(InMemoryOffsetStore::new());

    let mut manager = OffsetManager::new(
        "my-consumer-group".to_string(),
        backend,
        store,
        OffsetCommitStrategy::Periodic {
            interval: Duration::from_secs(5),
        },
    );

    // Update offset after processing event
    manager.update_offset("my-topic", 0, 100).await?;

    // Commit offsets
    manager.commit().await?;

    // Check lag
    let lag = manager.lag("my-topic", 0, 200).await?;
    println!("Current lag: {}", lag);

    Ok(())
}

Structs

InMemoryOffsetStore

In-memory offset store implementation

OffsetInfo

Offset information for a partition

OffsetManager

Main offset manager

OffsetStats

Statistics about offset management

StateBackendOffsetStore

State backend offset store implementation

TopicPartition

Represents a topic-partition pair

Enums

OffsetCommitStrategy

Strategy for committing offsets

OffsetResetStrategy

Reset strategy for partition offsets

Traits

OffsetStore

Trait for offset storage backends