Module concurrent_pipeline 

Concurrent document processing pipeline for parallel GraphRAG operations.

This module provides the ConcurrentProcessor for processing multiple documents in parallel while respecting concurrency limits and coordinating with rate limiters and metrics tracking.

Main Types

• ConcurrentProcessor: Manages concurrent document processing with configurable parallelism limits

Features

• Concurrent processing of document batches with configurable limits
• Automatic chunking to respect concurrency constraints
• Integration with rate limiting to prevent API throttling
• Comprehensive metrics tracking for all operations
• Error isolation: failures in one document don’t affect others
• Task spawning with tokio for true parallel execution

Basic Usage

use graphrag_core::async_processing::ConcurrentProcessor;
use std::sync::Arc;

let processor = ConcurrentProcessor::new(10); // Max 10 concurrent documents

let results = processor.process_batch(
    documents,
    graph,
    rate_limiter,
    metrics
).await?;

for result in results {
    println!("Processed document {} in {:?}",
        result.document_id,
        result.processing_time
    );
}

Structs

ConcurrentProcessor

Concurrent document processor for parallel GraphRAG operations