Module progress_indicator 

Progress Indicator Service

This module provides a real-time progress indicator for user feedback during pipeline processing operations. It offers immediate visual feedback to users about processing progress, separate from logging and metrics systems.

Overview

The progress indicator service provides:

• Real-Time Updates: Live progress updates as chunks are processed
• User-Focused Feedback: Immediate visual feedback for end users
• Terminal Integration: Direct terminal output with in-place updates
• Thread Safety: Concurrent-safe for multi-threaded processing
• Performance Metrics: Throughput and timing information

Architecture

The progress indicator follows these design principles:

┌─────────────────────────────────────────────────────────────────┐
│                    Progress Indicator System                     │
│                                                                     │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │                 Real-Time Display                       │    │
│  │  - Terminal output with in-place updates                │    │
│  │  - Chunk progress tracking                              │    │
│  │  - Throughput calculations                              │    │
│  └─────────────────────────────────────────────────────────┘    │
│                                                                     │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │                Thread-Safe Tracking                    │    │
│  │  - Atomic counters for concurrent access               │    │
│  │  - Mutex coordination for terminal output              │    │
│  │  - Lock-free progress updates                           │    │
│  └─────────────────────────────────────────────────────────┘    │
│                                                                     │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │              Performance Monitoring                   │    │
│  │  - Throughput calculation (MB/s)                       │    │
│  │  - Duration tracking                                    │    │
│  │  - Completion statistics                               │    │
│  └─────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────┘

Design Principles

User-Focused Design

The progress indicator is designed specifically for end-user feedback:

• Immediate Feedback: Updates appear as soon as chunks are processed
• Clear Format: Easy-to-read progress format with chunk IDs and counts
• Non-Intrusive: Doesn’t interfere with normal application logging
• Terminal Integration: Works seamlessly with terminal-based applications

Separation of Concerns

Progress indication is separate from other monitoring systems:

• Not Logging: Writes directly to terminal, bypassing logging systems
• Not Metrics: Focused on user feedback, not system monitoring
• Real-Time: Updates immediately, not batched or aggregated
• Ephemeral: Progress display is temporary and contextual

Performance Considerations

• Minimal Overhead: Lightweight implementation to avoid performance impact
• Atomic Operations: Lock-free progress updates using atomic counters
• Coordinated Output: Mutex only for terminal output coordination
• Efficient Updates: In-place terminal updates without scrolling

Output Format

Progress Display

The progress indicator shows real-time chunk processing status:

Wrote Id: 000097/Completed: 002000

• Wrote Id: Last chunk ID that was processed
• Completed: Total number of chunks completed
• Format: Zero-padded for consistent alignment

Completion Summary

Upon completion, shows comprehensive statistics:

✅ Processing completed successfully!
📄 Processed: 1.25 GB
⏱️ Duration: 2.34 seconds
🚀 Throughput: 534.2 MB/s
📊 Chunks: 2000 total

Usage Examples

Basic Progress Tracking

Concurrent Processing

Integration with File Processing

Thread Safety

The progress indicator is designed for concurrent use:

Atomic Counters

• Completed Chunks: Atomic counter for lock-free updates
• Last Chunk ID: Atomic storage for the most recent chunk ID
• Performance: No contention on progress updates

Terminal Coordination

• Output Mutex: Coordinates terminal output to prevent garbled display
• Minimal Locking: Mutex only held during actual terminal writes
• Non-Blocking: Progress updates don’t block on terminal output

Performance Impact

The progress indicator is designed to have minimal performance impact:

• Atomic Operations: Lock-free progress updates
• Minimal Allocations: Reuses buffers and avoids unnecessary allocations
• Efficient Terminal I/O: Direct terminal writes without buffering
• Optional: Can be disabled in production environments if needed

Integration with Other Systems

Logging System

Progress indication is separate from logging:

• No Log Interference: Doesn’t interfere with structured logging
• Direct Terminal: Writes directly to terminal, not through log handlers
• Complementary: Works alongside logging for different purposes

Metrics System

Progress indication complements metrics collection:

• Different Purpose: User feedback vs. system monitoring
• Real-Time: Immediate updates vs. aggregated metrics
• Ephemeral: Temporary display vs. persistent metrics storage

Error Handling

The progress indicator handles errors gracefully:

• Terminal Errors: Gracefully handles terminal I/O errors
• Non-Fatal: Progress indicator failures don’t affect processing
• Fallback: Can fall back to silent operation if terminal is unavailable
• Recovery: Automatically recovers from transient terminal issues

Structs

ProgressIndicatorService

Real-time progress indicator for user feedback during pipeline processing.