pub struct ConcurrentProcessor { /* private fields */ }Expand description
Concurrent document processor for parallel GraphRAG operations
Manages concurrent processing of multiple documents while respecting concurrency limits and coordinating with rate limiters and metrics tracking.
Implementations§
Source§impl ConcurrentProcessor
impl ConcurrentProcessor
Sourcepub fn new(max_concurrent_documents: usize) -> Self
pub fn new(max_concurrent_documents: usize) -> Self
Creates a new concurrent processor with specified concurrency limit
§Parameters
max_concurrent_documents: Maximum number of documents to process in parallel
Sourcepub async fn process_batch(
&self,
documents: Vec<Document>,
graph: Arc<RwLock<KnowledgeGraph>>,
rate_limiter: Arc<RateLimiter>,
metrics: Arc<ProcessingMetrics>,
) -> Result<Vec<ProcessingResult>, GraphRAGError>
pub async fn process_batch( &self, documents: Vec<Document>, graph: Arc<RwLock<KnowledgeGraph>>, rate_limiter: Arc<RateLimiter>, metrics: Arc<ProcessingMetrics>, ) -> Result<Vec<ProcessingResult>, GraphRAGError>
Processes a batch of documents concurrently with rate limiting and metrics tracking
Documents are processed in chunks according to the concurrency limit. Each chunk is fully processed before moving to the next, with a small delay between chunks to prevent system overload.
§Parameters
documents: Collection of documents to processgraph: Shared knowledge graph for storing extracted entitiesrate_limiter: Rate limiter for API call throttlingmetrics: Metrics collector for tracking processing statistics
§Returns
Vector of processing results for successfully processed documents, or an error
Trait Implementations§
Auto Trait Implementations§
impl Freeze for ConcurrentProcessor
impl RefUnwindSafe for ConcurrentProcessor
impl Send for ConcurrentProcessor
impl Sync for ConcurrentProcessor
impl Unpin for ConcurrentProcessor
impl UnsafeUnpin for ConcurrentProcessor
impl UnwindSafe for ConcurrentProcessor
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
Source§impl<T> Instrument for T
impl<T> Instrument for T
Source§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
Source§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more