Module plugins

Expand description

Plugin system for extending Xberg functionality.

The plugin system provides a trait-based architecture that allows extending Xberg with custom extractors, OCR backends, post-processors, and validators.

§Plugin Types

Plugin - Base trait that all plugins must implement
OcrBackend - OCR processing plugins
EmbeddingBackend - In-process embedding backend plugins
DocumentExtractor - Document format extraction plugins
PostProcessor - Content post-processing plugins
Validator - Validation plugins

§Language Support

Plugins can be implemented in:

Rust (native, highest performance)
Python (via PyO3 FFI bridge)
Node.js (future - via napi-rs FFI bridge)

§Lifecycle Pattern

Plugins are stored in Arc<dyn Trait> for thread-safe shared access:

use xberg::plugins::{Plugin, DocumentExtractor};
use xberg::plugins::registry::get_document_extractor_registry;
use std::sync::Arc;

// 1. Create plugin instance
let plugin = MyExtractor;

// 2. Wrap in Arc for registration
let plugin = Arc::new(plugin);

// 3. Register with registry (calls initialize internally)
let registry = get_document_extractor_registry();
let mut registry = registry.write();
registry.register(plugin)?;

§Example: Custom Document Extractor

use xberg::plugins::{Plugin, DocumentExtractor};
use xberg::{ExtractInput, ExtractionConfig, Result};
use xberg::types::{ExtractedDocument, Metadata};
use async_trait::async_trait;

struct CustomJsonExtractor;

impl Plugin for CustomJsonExtractor {
    fn name(&self) -> &str { "custom-json-extractor" }
    fn version(&self) -> String { "1.0.0".to_string() }
    fn initialize(&self) -> Result<()> {
        println!("JSON extractor initialized");
        Ok(())
    }
    fn shutdown(&self) -> Result<()> {
        println!("JSON extractor shutdown");
        Ok(())
    }
}

#[async_trait]
impl DocumentExtractor for CustomJsonExtractor {
    async fn extract(&self, input: ExtractInput, _config: &ExtractionConfig)
        -> Result<ExtractedDocument> {
        // Parse JSON and extract all string values
        let content = input.bytes.unwrap_or_default();
        let json: serde_json::Value = serde_json::from_slice(&content)?;
        let extracted_text = extract_strings_from_json(&json);

        let mut metadata = Metadata::default();
        metadata.additional.insert("extracted_fields".to_string().into(), serde_json::json!(true));

        // `ExtractedDocument` has private internal fields, so a struct literal with
        // `..Default::default()` does not compile outside the crate. Build a default
        // and assign the public fields instead.
        let mut document = ExtractedDocument::default();
        document.content = extracted_text;
        document.mime_type = std::borrow::Cow::Borrowed("application/json");
        document.metadata = metadata;
        Ok(document)
    }

    fn supported_mime_types(&self) -> &[&str] {
        &["application/json", "text/json"]
    }

    fn priority(&self) -> i32 { 50 } // Default priority
}

fn extract_strings_from_json(value: &serde_json::Value) -> String {
    match value {
        serde_json::Value::String(s) => format!("{}\n", s),
        serde_json::Value::Array(arr) => {
            arr.iter().map(extract_strings_from_json).collect()
        }
        serde_json::Value::Object(obj) => {
            obj.values().map(extract_strings_from_json).collect()
        }
        _ => String::new(),
    }
}

§Safety and Threading

CRITICAL: All plugins must be Send + Sync because they are:

Stored in Arc<dyn Trait> for shared ownership
Accessed concurrently from multiple threads
Called with &self (shared references)

Interior Mutability Pattern: Since plugins receive &self (not &mut self), use these for mutable state:

Mutex<T> - Exclusive access, blocking
RwLock<T> - Shared read, exclusive write
AtomicBool / AtomicU64 - Lock-free primitives
OnceCell<T> - One-time initialization

use xberg::plugins::Plugin;
use std::sync::Mutex;

struct StatefulPlugin {
    // Use interior mutability for state
    call_count: std::sync::atomic::AtomicU64,
    cache: Mutex<Option<Vec<String>>>,
}

impl Plugin for StatefulPlugin {
    fn name(&self) -> &str { "stateful-plugin" }
    fn version(&self) -> String { "1.0.0".to_string() }

    fn initialize(&self) -> xberg::Result<()> {
        // Modify through interior mutability
        let mut cache = self.cache.lock().unwrap();
        *cache = Some(vec!["initialized".to_string()]);
        Ok(())
    }

    fn shutdown(&self) -> xberg::Result<()> {
        self.call_count.store(0, std::sync::atomic::Ordering::Release);
        Ok(())
    }
}

Re-exports§

pub use processor::PostProcessor;
pub use processor::ProcessingStage;
pub use processor::clear_post_processors;
pub use processor::list_post_processors;
pub use processor::register_post_processor;
pub use processor::unregister_post_processor;
pub use renderer::Renderer;
pub use renderer::clear_renderers;
pub use renderer::list_renderers;
pub use renderer::register_renderer;
pub use renderer::unregister_renderer;
pub use validator::Validator;
pub use validator::clear_validators;
pub use validator::list_validators;
pub use validator::register_validator;
pub use validator::unregister_validator;

Modules§

document_extractor: Re-exports for the document extractor plugin type, used by alef-generated bindings.
embedding_backend: Re-exports for the embedding backend plugin type, used by alef-generated bindings.
ocr_backend: Re-exports for the OCR backend plugin type, used by alef-generated bindings.
post_processor: Re-exports for the post-processor plugin type, used by alef-generated bindings.
processor: Post-processor plugin system.
registry: Plugin registration and discovery.
renderer: Renderer plugin trait.
reranker_backend: Re-exports for the reranker backend plugin type, used by alef-generated bindings.
startup_validation: Startup validation for plugin registries.
tokenizer_backend: Re-exports for the tokenizer backend plugin type, used by alef-generated bindings.
validator: Validator plugin system.

Structs§

OcrBackendCapabilities: A registered OCR backend’s declared name and language capabilities.

Enums§

ConfidenceSemantics: How a backend’s reported page-level confidence must be interpreted.
OcrBackendType: OCR backend types.
PageOrientationHandling: How a backend copes with a page raster whose text is not upright.

Traits§

DocumentExtractor: Trait for document extractor plugins.
EmbeddingBackend: Trait for in-process embedding backend plugins.
InternalDocumentExtractor: Low-level extraction capability used by native Rust extractors.
OcrBackend: Trait for OCR backend plugins.
Plugin: Base trait that all plugins must implement.
RerankerBackend: Trait for in-process reranker backend plugins.
TokenizerBackend: Trait for in-process tokenizer backend plugins.

Functions§

clear_document_extractors: Clear all document extractors from the global registry.
clear_embedding_backends: Clear all embedding backends from the global registry.
clear_ocr_backends: Clear all OCR backends from the global registry.
clear_reranker_backends: Clear all reranker backends from the global registry.
clear_tokenizer_backends: Clear all tokenizer backends from the global registry.
list_document_extractors: List names of all registered document extractors.
list_embedding_backends: List the names of all registered embedding backends.
list_ocr_backend_capabilities: List every registered OCR backend’s name alongside its declared supported languages.
list_ocr_backend_capabilities_for: list_ocr_backend_capabilities, reporting each backend’s languages under config.
list_ocr_backends: List all registered OCR backends.
list_reranker_backends: List the names of all registered reranker backends.
list_tokenizer_backends: List the names of all registered tokenizer backends.
ocr_backend_supports_language: Check whether a specific registered OCR backend supports a language.
ocr_backend_supports_language_for: ocr_backend_supports_language, answering under config.
register_document_extractor: Register a document extractor with the global registry.
register_embedding_backend: Register an embedding backend with the global registry.
register_ocr_backend: Register an OCR backend with the global registry.
register_reranker_backend: Register a reranker backend with the global registry.
register_tokenizer_backend: Register a tokenizer backend with the global registry.
unregister_document_extractor: Unregister a document extractor by name.
unregister_embedding_backend: Unregister an embedding backend by name, calling its shutdown() method.
unregister_ocr_backend: Unregister an OCR backend by name.
unregister_reranker_backend: Unregister a reranker backend by name, calling its shutdown() method.
unregister_tokenizer_backend: Unregister a tokenizer backend by name, calling its shutdown() method.

Module plugins

Module plugins Copy item path

§Plugin Types

§Language Support

§Lifecycle Pattern

§Example: Custom Document Extractor

§Safety and Threading

Re-exports§

Modules§

Structs§

Enums§

Traits§

Functions§

Module plugins