Trait Preprocessor

Source

pub trait Preprocessor: Send + Sync {
    // Required methods
    fn parse_pdf_to_markup_language(
        &self,
        pdf_bytes: &[u8],
    ) -> Result<String, Error>;
    fn parse_markup_to_preprocessor_output(
        &self,
        markup: &str,
    ) -> Result<PreprocessorOutput, Error>;
    fn name(&self) -> &str;
    fn supports_file_type(&self, path: &Path) -> bool;

    // Provided methods
    fn process(
        &self,
        document_bytes: &[u8],
    ) -> Result<PreprocessorOutput, Error> { ... }
    fn process_file(&self, input: &Path) -> Result<PreprocessorOutput, Error> { ... }
}

Expand description

Preprocessor trait - converts documents to PreprocessorOutput

This is the key abstraction boundary in blazegraph. Preprocessors handle:

Document format parsing (PDF, Word, etc.)
Text extraction and positioning
Basic structure detection (pages, paragraphs, etc.)

Everything after this point works with PreprocessorOutput and is format-agnostic.

The preprocessing happens in two clear steps:

Document -> Markup Language (e.g., PDF -> XHTML)
Markup Language -> PreprocessorOutput (structured data)

Required Methods§

Source

fn parse_pdf_to_markup_language( &self, pdf_bytes: &[u8], ) -> Result<String, Error>

Step 1: Convert document to markup language

For PDF: PDF bytes -> XHTML For other preprocessors: DOC bytes -> HTML, etc. This step handles the raw document format conversion.

Source

fn parse_markup_to_preprocessor_output( &self, markup: &str, ) -> Result<PreprocessorOutput, Error>

Step 2: Convert markup language to structured output

Parses markup (XHTML, HTML, etc.) into our structured format with text elements, metadata, styling, and bookmarks. This step is format-agnostic after step 1.

Source