Struct CrossEncoderEngine 

pub struct CrossEncoderEngine { /* private fields */ }

Cross-encoder reranking engine.

Thread-safe — shared via Arc. Maintains a pool of independent ONNX sessions so concurrent rerank calls never contend on a single mutex.

Implementations

impl CrossEncoderEngine

pub async fn new(cache_dir: Option<String>) -> Result<Self>

Load or download the reranker model.

Downloads Xenova/bge-reranker-base ONNX INT8 model from HuggingFace Hub if not already cached. Builds RERANKER_POOL_SIZE independent sessions.

pub async fn score_pairs( &self, query: &str, passages: &[String], ) -> Result<Vec<f32>>

Score a batch of (query, passage) pairs.

Passages are split into chunks of [RERANKER_CHUNK_SIZE] and dispatched in parallel across the session pool (round-robin). Within each chunk, pairs are processed in mini-batches of [RERANKER_ONNX_BATCH_SIZE] to reduce sequence-padding overhead. Chunk results are reassembled in input order.

Returns Err(InferenceError::Overloaded) immediately when more than RERANKER_MAX_CONCURRENT callers are active — the API layer falls back to unranked results rather than queuing indefinitely (DAK-5893 fix).

Returns a relevance score in [0, 1] for each passage. Higher scores indicate greater relevance to the query.

pub fn pool_size(&self) -> usize

Number of parallel ONNX sessions in the pool.

pub fn onnx_batch_size(&self) -> usize

Configured ONNX mini-batch size (pairs per session.run() call).

pub fn active_requests_count(&self) -> usize

Current number of active concurrent score_pairs calls. Used by metrics and health checks (DAK-5893).

pub fn max_concurrent(&self) -> usize

Maximum concurrent score_pairs calls before Overloaded is returned.

Trait Implementations

impl Debug for CrossEncoderEngine

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.