# Usage Guide
This guide covers the detailed usage of OAROCR for text recognition and document structure analysis.
## Basic OCR Pipeline
### Simple Usage
The `pp_ocrv6` preset selects the models, dictionary, and official detection thresholds for a PP-OCRv6 size, so the pipeline is one line. With the `auto-download` feature the names resolve through the model registry; without it they resolve as local paths, like any other model source.
```rust
use oar_ocr::prelude::*;
use std::path::Path;
fn main() -> Result<(), Box<dyn std::error::Error>> {
// Build a PP-OCRv6 pipeline at the chosen size
let ocr = OAROCRBuilder::pp_ocrv6(PpOcrV6Size::Small).build()?;
// Process a single image
let image = load_image(Path::new("document.jpg"))?;
let results = ocr.predict(vec![image])?;
let result = &results[0];
// Print extracted text with confidence scores
for text_region in &result.text_regions {
if let Some((text, confidence)) = text_region.text_with_confidence() {
println!("Text: {} (confidence: {:.2})", text, confidence);
}
}
Ok(())
}
```
The explicit form, for choosing every model yourself:
```rust
use oar_ocr::prelude::*;
use std::path::Path;
fn main() -> Result<(), Box<dyn std::error::Error>> {
// Build OCR pipeline with required models
let ocr = OAROCRBuilder::new(
"pp-ocrv5_mobile_det.onnx",
"pp-ocrv5_mobile_rec.onnx",
"ppocrv5_dict.txt",
)
.build()?;
// Process a single image
let image = load_image(Path::new("document.jpg"))?;
let results = ocr.predict(vec![image])?;
let result = &results[0];
// Print extracted text with confidence scores
for text_region in &result.text_regions {
if let Some((text, confidence)) = text_region.text_with_confidence() {
println!("Text: {} (confidence: {:.2})", text, confidence);
}
}
Ok(())
}
```
### Batch Processing
```rust
// Process multiple images at once (accepts &str paths)
let images = load_images(&[
"document1.jpg",
"document2.jpg",
"document3.jpg",
])?;
let results = ocr.predict(images)?;
for result in results {
println!("Image {}: {} text regions found", result.index, result.text_regions.len());
for text_region in &result.text_regions {
if let Some((text, confidence)) = text_region.text_with_confidence() {
println!(" Text: {} (confidence: {:.2})", text, confidence);
}
}
}
```
## Builder APIs
OAROCR provides two high-level builder APIs for easy pipeline construction.
### OAROCRBuilder - Text Recognition Pipeline
The `OAROCRBuilder` provides a fluent API for building OCR pipelines with optional components:
```rust
use oar_ocr::oarocr::OAROCRBuilder;
// Basic OCR pipeline
let ocr = OAROCRBuilder::new(
"pp-ocrv5_mobile_det.onnx",
"pp-ocrv5_mobile_rec.onnx",
"ppocrv5_dict.txt",
)
.build()?;
// OCR with optional preprocessing
let ocr = OAROCRBuilder::new(
"pp-ocrv5_mobile_det.onnx",
"pp-ocrv5_mobile_rec.onnx",
"ppocrv5_dict.txt",
)
.with_document_image_orientation_classification("pp-lcnet_x1_0_doc_ori.onnx")
.with_text_line_orientation_classification("pp-lcnet_x1_0_textline_ori.onnx")
.with_document_image_rectification("uvdoc.onnx")
.image_batch_size(4)
.region_batch_size(64)
.build()?;
```
#### Available Options
| Method | Description |
|--------|-------------|
| `.with_document_image_orientation_classification(path)` | Add document orientation detection |
| `.with_text_line_orientation_classification(path)` | Add text line orientation detection |
| `.with_document_image_rectification(path)` | Add document rectification (UVDoc) |
| `.text_type("seal")` | Optimize pipeline for curved seal/stamp text |
| `.return_word_box(true)` | Enable word-level bounding boxes |
| `.image_batch_size(n)` | Set batch size for image processing |
| `.region_batch_size(n)` | Set batch size for region processing |
| `.ort_session(config)` | Apply ONNX Runtime configuration |
| `.character_dict_content(text)` | Provide the character dictionary as an in-memory string |
#### Loading Models from Memory
Everywhere a builder accepts a model path it also accepts raw ONNX bytes (`Vec<u8>`, `&'static [u8]`, or `Arc<[u8]>`), so models can be embedded into the binary or decrypted at runtime without touching the filesystem:
```rust
use oar_ocr::oarocr::OAROCRBuilder;
static DET_MODEL: &[u8] = include_bytes!("pp-ocrv6_tiny_det.onnx");
static REC_MODEL: &[u8] = include_bytes!("pp-ocrv6_tiny_rec.onnx");
static DICT: &str = include_str!("ppocrv6_tiny_dict.txt");
let ocr = OAROCRBuilder::new(DET_MODEL, REC_MODEL, "")
.character_dict_content(DICT)
.build()?;
```
In-memory sources skip auto-download resolution, and models that reference external-data sidecar files cannot be loaded this way. The same applies to the per-task predictors (`TextDetectionPredictorBuilder::build(...)` etc.), `OARStructureBuilder` model setters, and `AdapterBuilder::build(...)`.
### OARStructureBuilder - Document Structure Analysis
The `OARStructureBuilder` enables document structure analysis with layout detection, table recognition, and formula extraction. The `pp_structurev3` preset wires the full PP-StructureV3-style stack in one call — PP-DocLayoutV3 layout, PP-OCRv6 Tiny OCR, table classification, SLANeXt wired and SLANet+ wireless table structure, wired cell detection, and the table dictionary:
```rust
use oar_ocr::oarocr::OARStructureBuilder;
// Full PP-StructureV3-style document analysis in one call
let structure = OARStructureBuilder::pp_structurev3().build()?;
```
Each component can also be selected explicitly to build a custom pipeline, for example with a single wired table model:
```rust
use oar_ocr::oarocr::OARStructureBuilder;
// Basic layout detection
let structure = OARStructureBuilder::new("picodet-l_layout_17cls.onnx")
.build()?;
// Full document structure analysis
let structure = OARStructureBuilder::new("picodet-l_layout_17cls.onnx")
.with_table_classification("pp-lcnet_x1_0_table_cls.onnx")
.with_table_cell_detection("rt-detr-l_wired_table_cell_det.onnx", "wired")
.with_table_structure_recognition("slanext_wired.onnx", "wired")
.table_structure_dict_path("table_structure_dict_ch.txt")
.with_formula_recognition(
"pp-formulanet-l.onnx",
"pp-formulanet-tokenizer.json",
"pp_formulanet",
)
.build()?;
// Structure analysis with integrated OCR
let structure = OARStructureBuilder::new("picodet-l_layout_17cls.onnx")
.with_table_classification("pp-lcnet_x1_0_table_cls.onnx")
.with_ocr("pp-ocrv5_mobile_det.onnx", "pp-ocrv5_mobile_rec.onnx", "ppocrv5_dict.txt")
.build()?;
```
#### Available Options
| Method | Description |
|--------|-------------|
| `.with_table_classification(path)` | Add wired/wireless table classification |
| `.with_table_cell_detection(path, type)` | Add table cell detection |
| `.with_table_structure_recognition(path, type)` | Add table structure recognition |
| `.table_structure_dict_path(path)` | Set table structure dictionary |
| `.with_formula_recognition(model, tokenizer, type)` | Add formula recognition |
| `.formula_recognition_config(config)` | Set formula score threshold, max length, and batch size |
| `.formula_ort_session(config)` | Apply ONNX Runtime configuration only to formula recognition |
| `.with_ocr(det, rec, dict)` | Add integrated OCR pipeline |
| `.with_seal_text_detection(path)` | Add seal/stamp text detection |
| `.image_batch_size(n)` | Set batch size for image processing |
| `.region_batch_size(n)` | Set batch size for region processing |
| `.ort_session(config)` | Apply ONNX Runtime configuration |
### IndraOcrBuilder - Indra OCR
[Indra OCR](https://huggingface.co/datalab-to/indra-ocr) is Datalab's CPU-first page reader: small ONNX models (about 76 MB, 46 MB of them n-gram language models for lines a recognizer is unsure of) that turn a page into lines with boxes, layout blocks in reading order (text, section headers, lists, tables, captions, footnotes, page headers and footers), and Markdown with tables as HTML. A script classifier sends each line to one of four recognizers: Latin, Cyrillic, and Greek (including handwriting); Chinese, Japanese, and Korean; Arabic, Persian, Urdu, and Hebrew (read into logical order, with right-to-left pages read right column first); and Indic, Thai, Lao, Khmer, Georgian, Armenian, and other scripts. It turns quarter-turned pages upright, straightens tilts up to 45 degrees, reads light text on dark ground, and reads Chinese and Japanese pages set in vertical columns. `oar_ocr::indra` reproduces the reference pipeline's OCR mode (`indra-ocr` 1.1.2 with the v1.1.0 models; the earlier v3.1 model set still loads), with two fixes: token positions from the tall-window recognizer are no longer stretched, which moves some split-line boxes, and tables built from the table heads no longer drop a cell that a neighbour's span would cover. Reading a PDF's own text layer is not supported, so render PDF pages to images first. Download the models from Hugging Face; they are not part of the auto-download registry, and their weights use a modified OpenRAIL-M license.
The [`oar indra` command](../oar-ocr-cli/README.md#indra-ocr) downloads its checkpoint directly from Hugging Face and accepts images or rasterized PDF pages. Use `--model-dir` for an existing local checkpoint.
```bash
oar indra document.pdf --device cpu --workers 4 --pages 1-8 -o pages
oar indra page.png --device cuda:0 --format json
```
```bash
hf download datalab-to/indra-ocr --local-dir models/datalab-to/indra-ocr
```
```rust,no_run
use oar_ocr::indra::IndraOcrBuilder;
use oar_ocr::utils::load_image;
let ocr = IndraOcrBuilder::new("models/datalab-to/indra-ocr")
.math(true) // write lines with math as inline LaTeX
.build()?;
let page = ocr.predict(vec![load_image("page.png")?])?.remove(0);
println!("{}", page.markdown);
for block in &page.blocks {
println!("{} {:?}", block.label, block.bbox);
}
let json = page.to_json(); // the shared page JSON, with pipeline "indra"
# Ok::<(), Box<dyn std::error::Error>>(())
```
| Method | Description |
|--------|-------------|
| `.math(bool)` | Read lines with math as text with inline LaTeX (off by default; needs the `math_*` model files) |
| `.keep_furniture(bool)` | Keep page headers and footers in the Markdown (left out by default; always listed in `blocks`) |
| `.table_heads(bool)` | Build tables from the order model's row, column and cell heads instead of from line geometry (off by default, as in the reference; a table the heads fail on falls back to geometry) |
| `.workers(usize)` | CPU page workers (default: 1); each has its own ONNX sessions and shares vocabularies and language models |
| `.ort_session(config)` | Apply ONNX Runtime configuration to every model |
Pass several images to one `predict` call to use the workers; up to the configured number of pages are read concurrently, and pages and errors retain input order. With multiple workers, unset ONNX intra/inter-op thread counts default to one. Explicit thread settings are honored; keep workers times threads within the available cores. Per-page preprocessing uses those worker threads rather than the global pool, so a small worker count also limits preprocessing parallelism. An application-configured global ONNX Runtime thread pool retains its process-wide settings. The default single-worker path is unchanged.
On an Intel Core i9-13900KF (8 performance cores and 16 efficiency cores), warmed runs with the v1.1.0 models, ONNX intra/inter-op threads set to one, and optional math and table heads disabled measured 1.62 pages/s with one worker on 48 pages. Repeating those pages six times (288 pages) measured 7.9 pages/s with 8 workers, 8.6 with 16, and 8.5 with 32. Throughput stopped rising beyond the 16 performance-core threads in this test; start with workers equal to the number of performance-core threads and measure on your own hardware.
Memory grows with the number of workers: peak RSS on the 48-page set rose from about 3 GB with 1 worker to about 5 GB with 16. Each worker owns model sessions, activation arenas, and temporary page buffers, while vocabularies and n-gram language models are shared.
For CUDA, build with `--features cuda` and select the CUDA execution provider through `.ort_session(config)`, as in [GPU Acceleration](#gpu-acceleration). Accelerator configurations require `.workers(1)`; larger values return a configuration error. The example accepts `--device cuda:0`. On an RTX 4090, the 48-page set reached 2.6 pages/s, about 1.6 times one CPU worker, with about 13 GB of VRAM; several CPU workers were faster on this workload. Recognized text matched CPU, but near-tie reading-order or table-column decisions can differ on a few pages.
```bash
cargo run --release --example indra -- --model-dir models/datalab-to/indra-ocr --workers 4 page1.png page2.png page3.png page4.png
cargo run --release --features cuda --example indra -- --model-dir models/datalab-to/indra-ocr --device cuda:0 page.png
```
Boxes are in the upright page: `page.rotation` is the quarter turn that was undone and `page.width` / `page.height` its size, and `page.skew` the tilt that was straightened (boxes are mapped back across it). A page set in vertical columns has `page.vertical` set; it is read turned a quarter turn, but its boxes are on the page as it is and `page.rotation` stays 0.
## GPU Acceleration
### Automatic Device Selection
Examples default to `--device auto` (case-insensitive). Classic pipelines try compiled CUDA(0), CoreML on Apple platforms, and DirectML(0) on Windows, then CPU. Library callers opt in with `OrtSessionConfig::auto()`; existing defaults remain CPU, and `resolve_auto()` explicitly queries the resolved configuration. Failed registrations are omitted during construction, before choosing CPU batch defaults. TensorRT, OpenVINO, and WebGPU remain explicit choices because of initialization cost and compatibility. When CUDA is compiled but no GPU is available, ONNX Runtime prints an ERROR log during probing: `CUDA failure 100: no CUDA-capable device is detected`. This is expected; execution then falls back to CPU.
VL tries compiled CUDA(0), then Metal(0), then CPU. Use `oar_ocr_vl::auto_device()` or `oar_ocr_vl::utils::parse_device("auto")` in library code. Device creation failures fall back automatically; the existing dtype probe selects F32 on CPU and on accelerators without BF16 support, BF16 when the device supports it and a BF16 probe succeeds, or F16 when that probe fails. Use `--device cpu` to require CPU execution in either pipeline.
### CUDA
Enable CUDA support for GPU inference:
```rust
use oar_ocr::prelude::*;
use oar_ocr::core::config::{OrtSessionConfig, OrtExecutionProvider};
use std::path::Path;
fn main() -> Result<(), Box<dyn std::error::Error>> {
// Configure CUDA execution provider
let ort_config = OrtSessionConfig::new()
.with_execution_providers(vec![
OrtExecutionProvider::CUDA {
device_id: Some(0),
gpu_mem_limit: None,
arena_extend_strategy: None,
cudnn_conv_algo_search: None,
cudnn_conv_use_max_workspace: None,
},
OrtExecutionProvider::CPU, // Fallback
]);
// Build OCR pipeline with CUDA
let ocr = OAROCRBuilder::new(
"pp-ocrv5_mobile_det.onnx",
"pp-ocrv5_mobile_rec.onnx",
"ppocrv5_dict.txt",
)
.ort_session(ort_config)
.build()?;
// Use as normal
let image = load_image(Path::new("document.jpg"))?;
let results = ocr.predict(vec![image])?;
Ok(())
}
```
**Requirements:**
1. Install with CUDA feature: `cargo add oar-ocr --features cuda`
2. CUDA toolkit and cuDNN installed on your system
3. ONNX models compatible with CUDA execution
### Other Execution Providers
OAROCR supports multiple execution providers via feature flags:
See the [Cargo feature guide](features.md) for the root-crate features, default behavior, platform requirements, and recommended combinations.
| Feature | Provider | Platform |
|---------|----------|----------|
| `cuda` | NVIDIA CUDA | Linux, Windows |
| `tensorrt` | NVIDIA TensorRT | Linux, Windows |
| `directml` | DirectML | Windows |
| `coreml` | Core ML | macOS, iOS |
| `webgpu` | WebGPU | Supported ONNX Runtime targets |
| `openvino` | Intel OpenVINO | Linux, Windows |
Example with TensorRT:
```rust
let ort_config = OrtSessionConfig::new()
.with_execution_providers(vec![
OrtExecutionProvider::TensorRT {
device_id: Some(0),
max_workspace_size: None,
min_subgraph_size: None,
fp16_enable: None,
timing_cache: None,
timing_cache_path: None,
force_timing_cache: None,
engine_cache: None,
engine_cache_path: None,
dump_ep_context_model: None,
ep_context_file_path: None,
},
OrtExecutionProvider::CUDA {
device_id: Some(0),
gpu_mem_limit: None,
arena_extend_strategy: None,
cudnn_conv_algo_search: None,
cudnn_conv_use_max_workspace: None,
},
OrtExecutionProvider::CPU,
]);
```
## PaddleOCR-VL
[PaddleOCR-VL](https://huggingface.co/PaddlePaddle/PaddleOCR-VL) is a 0.9B document Vision-Language Model from the PaddlePaddle team. It supports 109 languages and task-specific recognition for text, tables, formulas, and charts.
This functionality is available in the separate `oar-ocr-vl` crate, using [Candle](https://github.com/huggingface/candle) for native Rust inference.
PaddleOCR-VL-1.5 and PaddleOCR-VL-1.6 are drop-in replacements via `PaddleOcrVl::from_dir`. Both add **text spotting** and **seal recognition** to the original model's task set.
### Installation
`oar-ocr-vl` is self-contained and runs on Candle, so it pulls in **no ONNX Runtime** and does not depend on `oar-ocr-core`:
```toml
[dependencies]
oar-ocr-vl = "0.10"
```
For GPU acceleration, enable CUDA:
```toml
[dependencies]
oar-ocr-vl = { version = "0.10", features = ["cuda"] }
```
On macOS, use the `metal` feature instead.
Layout-first parsing with `DocParser` uses `PpDocLayout`, a native Candle port of PP-DocLayoutV2/V3, so no extra dependency is required — only the checkpoint. `PpDocLayout::from_dir` reads `config.json` and `model.safetensors`, which are published in the `_safetensors` repositories:
```bash
git lfs install
git clone https://huggingface.co/PaddlePaddle/PP-DocLayoutV3_safetensors
# or the V2 checkpoint, which predicts reading order with a pointer network
git clone https://huggingface.co/PaddlePaddle/PP-DocLayoutV2_safetensors
```
The unsuffixed `PaddlePaddle/PP-DocLayoutV3` repository ships Paddle inference weights instead and cannot be loaded here. To source layout from somewhere else, implement `oar_ocr_vl::LayoutSource` and pass it to `DocParser::parse`.
### Downloading the Model
Download the PaddleOCR-VL model from Hugging Face:
```bash
# Recommended git download
git lfs install
git clone https://huggingface.co/PaddlePaddle/PaddleOCR-VL
# PaddleOCR-VL-1.5
git clone https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.5
# PaddleOCR-VL-1.6
git clone https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.6
# Or using hf
hf download PaddlePaddle/PaddleOCR-VL --local-dir PaddleOCR-VL
hf download PaddlePaddle/PaddleOCR-VL-1.5 --local-dir PaddleOCR-VL-1.5
hf download PaddlePaddle/PaddleOCR-VL-1.6 --local-dir PaddleOCR-VL-1.6
```
### Basic Usage
```rust,no_run
use oar_ocr_vl::utils::image::load_image;
use oar_ocr_vl::{PaddleOcrVl, PaddleOcrVlTask};
use oar_ocr_vl::utils::parse_device;
use std::path::Path;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let image = load_image(Path::new("document.png"))?;
let device = parse_device("cpu")?; // or "cuda", "cuda:0", "metal"
let vl = PaddleOcrVl::from_dir("PaddlePaddle/PaddleOCR-VL", device)?;
// Element-level OCR. The API is batch-oriented, so pass one task per image.
let result = vl
.generate(&[image], &[PaddleOcrVlTask::Ocr], 256)?
.into_iter()
.next()
.expect("one result")?;
println!("{result}");
Ok(())
}
```
PaddleOCR-VL-1.5 and PaddleOCR-VL-1.6 use the same API:
```rust,no_run
use oar_ocr_vl::utils::image::load_image;
use oar_ocr_vl::{PaddleOcrVl, PaddleOcrVlTask};
use oar_ocr_vl::utils::parse_device;
use std::path::Path;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let image = load_image(Path::new("seal.png"))?;
let device = parse_device("cpu")?;
// Use "PaddlePaddle/PaddleOCR-VL-1.5" here to load the 1.5 checkpoint instead.
let vl = PaddleOcrVl::from_dir("PaddlePaddle/PaddleOCR-VL-1.6", device)?;
let result = vl
.generate(&[image], &[PaddleOcrVlTask::Seal], 256)?
.into_iter()
.next()
.expect("one result")?;
println!("{result}");
Ok(())
}
```
### Running the Example
```bash
cargo run -p oar-ocr-vl --features cuda --example paddleocr_vl -- \
-m PaddlePaddle/PaddleOCR-VL --device cuda --task ocr document.jpg
cargo run -p oar-ocr-vl --features cuda --example paddleocr_vl -- \
-m PaddlePaddle/PaddleOCR-VL-1.5 --device cuda --task spotting spotting.jpg
cargo run -p oar-ocr-vl --features cuda --example paddleocr_vl -- \
-m PaddlePaddle/PaddleOCR-VL-1.6 --device cuda --task seal seal.jpg
```
### Supported Tasks
| Task | Description | Output Format |
|------|-------------|---------------|
| `PaddleOcrVlTask::Ocr` | Text recognition | Plain text |
| `PaddleOcrVlTask::Table` | Table structure recognition | HTML |
| `PaddleOcrVlTask::Formula` | Mathematical formula recognition | LaTeX |
| `PaddleOcrVlTask::Chart` | Chart understanding | Structured text |
| `PaddleOcrVlTask::Spotting` | Text spotting (localization + recognition) | Structured text |
| `PaddleOcrVlTask::Seal` | Seal recognition | Plain text |
## OvisOCR2
[OvisOCR2](https://huggingface.co/ATH-MaaS/OvisOCR2) is a 0.8B end-to-end page parser in the `oar-ocr-vl` crate. It turns each complete page into one Markdown document using its model-native path. It also implements `RecognitionBackend` for externally detected `DocParser` crops, although the native full-page path is preferred for complete pages.
### Downloading the Model
```bash
git lfs install
git clone https://huggingface.co/ATH-MaaS/OvisOCR2
# Or using hf
hf download ATH-MaaS/OvisOCR2 --local-dir OvisOCR2
```
### Official Input and Output Contract
The library uses the official prompt internally; callers do not need to provide it. The leading newline below is part of the prompt:
```text
Extract all readable content from the image in natural human reading order and output the result as a single Markdown document. For charts or images, represent them using an HTML image tag: <img src="images/bbox_{left}_{top}_{right}_{bottom}.jpg" />, where left, top, right, bottom are bounding box coordinates scaled to [0, 1000). Format formulas as LaTeX. Format tables as HTML: <table>...</table>. Transcribe all other text as standard Markdown. Preserve the original text without translation or paraphrasing.
```
Input pages are converted to RGB, resized with bicubic antialiasing, and aligned to the model's 32-pixel factor. The official runtime constrains image area to `448²` through `2880²` pixels. Output is reading-order Markdown with LaTeX formulas and HTML tables. Visual regions may be emitted as `<img src="images/bbox_left_top_right_bottom.jpg" />`, with coordinates scaled to `[0, 1000)`.
`OvisOcr2::parse` applies truncated-repeat cleanup and removes standalone visual-region image-tag blocks, matching the default official post-processing. Use `parse_with_image_tags(..., true)` or `generate` when those tags must be retained.
### Basic Usage
```rust,no_run
use oar_ocr_vl::utils::image::load_image;
use oar_ocr_vl::ovisocr2::DEFAULT_MAX_NEW_TOKENS;
use oar_ocr_vl::utils::parse_device;
use oar_ocr_vl::OvisOcr2;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let image = load_image("document.jpg")?;
let model = OvisOcr2::from_dir("ATH-MaaS/OvisOCR2", parse_device("cpu")?)?;
let markdown = model
.parse(&[image], DEFAULT_MAX_NEW_TOKENS)?
.into_iter()
.next()
.expect("one result")?;
println!("{markdown}");
Ok(())
}
```
The API is batch-oriented: pass multiple page images to `parse` and receive one result per page in the same order. The upstream generation limit, exported as `DEFAULT_MAX_NEW_TOKENS`, is 16,384.
### Running the Example
```bash
cargo run -p oar-ocr-vl --features cuda --example ovisocr2 -- \
--model-dir ATH-MaaS/OvisOCR2 \
--device cuda:0 \
document-1.jpg document-2.jpg
```
Add `--keep-image-tags` to retain visual-region image-tag references, or use `--max-tokens` to override the 16,384-token default. The example does not create the referenced bounding-box crop files.
## OvisOCR3
[OvisOCR3](https://huggingface.co/ATH-MaaS/OvisOCR3) reuses the OvisOCR2 engine with its own verbatim task prompts. The explicit ID `ATH-MaaS/OvisOCR3` selects Markdown parsing through `AnyPageParser` and `oar parse --model ATH-MaaS/OvisOCR3`. Load the typed model to select structured output:
```rust,no_run
use candle_core::Device;
use oar_ocr_vl::{OvisOcr3, OvisOcr3ParseMode, OvisOcr3ParseOptions, PageParser};
use oar_ocr_vl::utils::image::load_image;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let model = OvisOcr3::from_dir("ATH-MaaS/OvisOCR3", Device::Cpu)?;
let image = load_image("document.png")?;
let options = OvisOcr3ParseOptions::default().with_mode(OvisOcr3ParseMode::Structured);
let page = model.parse_page(&image, &options)?;
println!("{}", page.to_json(image.width(), image.height()));
Ok(())
}
```
Structured boxes refer to the image supplied to the model, including cropped inputs. Invalid or truncated JSON logs a warning and falls back to Markdown without blocks, retaining the raw text and a `structured_json` diagnostic. See the [crate guide](../oar-ocr-vl/README.md#ovisocr3) for coordinate conversion, task mappings, prompt constants, and post-processing options.
## OXR-1.0
[OXR-1.0](https://huggingface.co/SoMarkAI/OXR-1.0) recognizes regions supplied by PP-DocLayout through the explicit model ID `SoMarkAI/OXR-1.0`.
```rust,no_run
use candle_core::Device;
use oar_ocr_vl::{AnyPageParser, AnyPageParserLoadOptions, AnyPageParserModel, AnyPageParserOptions, PageParser};
use oar_ocr_vl::utils::image::load_image;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let parser = AnyPageParser::from_dir_with_options(
AnyPageParserModel::Oxr,
"SoMarkAI/OXR-1.0",
Device::Cpu,
&AnyPageParserLoadOptions::default()
.with_layout_dir("PaddlePaddle/PP-DocLayoutV3_safetensors"),
)?;
let page = parser.parse_page(&load_image("document.png")?, &AnyPageParserOptions::default())?;
println!("{}", page.markdown.unwrap_or_default());
Ok(())
}
```
The weights are under [SoMark's OXR License](https://huggingface.co/SoMarkAI/OXR-1.0/blob/main/LICENSE), Apache 2.0 plus commercial thresholds, an attribution duty, and a ban on competing hosted APIs; users must accept these terms before using the weights.
See the [crate guide](../oar-ocr-vl/README.md#oxr-10) for OXR tasks, PP-DocLayout integration, and generation options.
## MonkeyOCRv2-S/B-Parsing
[MonkeyOCRv2-S-Parsing](https://huggingface.co/zenosai/MonkeyOCRv2-S-Parsing) and [MonkeyOCRv2-B-Parsing](https://huggingface.co/zenosai/MonkeyOCRv2-B-Parsing) are 0.6B and 0.7B document parsers. They pair Monkey ViT-S and ViT-B vision encoders, respectively, with the same Qwen3-0.6B decoder. The native Candle implementation reads either checkpoint's dimensions from its configuration and supports the five official tasks: `Layout`, `EndToEnd`, `Text`, `Formula`, and `Table`.
### Downloading the Model
```bash
git lfs install
git clone https://huggingface.co/zenosai/MonkeyOCRv2-S-Parsing
# Or using hf
hf download zenosai/MonkeyOCRv2-S-Parsing --local-dir MonkeyOCRv2-S-Parsing
# Use MonkeyOCRv2-B-Parsing instead for the ViT-B variant
hf download zenosai/MonkeyOCRv2-B-Parsing --local-dir MonkeyOCRv2-B-Parsing
```
### Basic Usage
```rust,no_run
use oar_ocr_vl::utils::image::load_image;
use oar_ocr_vl::utils::parse_device;
use oar_ocr_vl::{MonkeyOcrV2, MonkeyOcrV2Task};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let image = load_image("document.jpg")?;
let model = MonkeyOcrV2::from_dir(
"zenosai/MonkeyOCRv2-S-Parsing",
parse_device("cuda:0")?,
)?;
let output = model
.generate(&[image], &[MonkeyOcrV2Task::EndToEnd], 10_000)?
.into_iter()
.next()
.expect("one result")?;
println!("{output}");
Ok(())
}
```
`EndToEnd` returns a Python/JSON-like reading-order list of `{bbox, label, content}` items; box coordinates are normalized to `[0, 1000]`. `Layout` returns `{bbox, label}` items and applies the official layout-pass minimum area of 1,003,520 pixels. `Table` emits OTSL, while `Formula` emits LaTeX. `generate_with_prompts` is available for custom instructions.
### Running the Example
```bash
cargo run -p oar-ocr-vl --features cuda --example monkeyocrv2 -- \
--model-dir zenosai/MonkeyOCRv2-S-Parsing \
--device cuda:0 \
--task end-to-end \
document.jpg
```
The other `--task` values are `layout`, `text`, `formula`, and `table`. The model also implements `RecognitionBackend` for external-layout crop recognition, although its native tasks are the preferred full-page path.
The same API and example accept `MonkeyOCRv2-B-Parsing`; select it by changing `--model-dir` to that checkpoint directory.
## HunyuanOCR 1.5
[HunyuanOCR 1.5](https://huggingface.co/tencent/HunyuanOCR) is a lightweight OCR expert VLM. It is available in the `oar-ocr-vl` crate and supports prompt-driven image-to-text OCR. `HunyuanOcr::from_dir` automatically detects 1.5 at the model repository root and remains compatible with archived 1.0 weights under `v1.0/`.
HunyuanOCR 1.5 inputs use the checkpoint's 16M-pixel budget (up to a 4K square input). The 2048 value in `vision_config.max_image_size` describes the learned positional-embedding base grid. Larger input grids are interpolated, as in the official implementation.
### Downloading the Model
```bash
git lfs install
git clone https://huggingface.co/tencent/HunyuanOCR
# Or using hf
hf download tencent/HunyuanOCR --local-dir HunyuanOCR
```
The download places 1.5 weights directly in `HunyuanOCR/` and its optional DFlash draft in `HunyuanOCR/dflash/`. To use 1.0 instead, pass `HunyuanOCR/v1.0` as the model directory.
### Basic Usage
```rust,no_run
use oar_ocr_vl::utils::image::load_image;
use oar_ocr_vl::HunyuanOcr;
use oar_ocr_vl::utils::parse_device;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let image = load_image("document.jpg")?;
let device = parse_device("cpu")?; // or "cuda", "cuda:0", "metal"
// Repository root = HunyuanOCR 1.5; `HunyuanOCR/v1.0` also works.
let model = HunyuanOcr::from_dir("tencent/HunyuanOCR", device)?;
let prompt = "Detect and recognize text in the image, and output the text coordinates in a formatted manner.";
let text = model
.generate(&[image], &[prompt], 1024)?
.into_iter()
.next()
.expect("one result")?;
println!("{text}");
Ok(())
}
```
HunyuanOCR 1.5 can load the official DFlash draft for speculative decoding:
```rust,no_run
use oar_ocr_vl::HunyuanOcr;
use oar_ocr_vl::utils::parse_device;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let model = HunyuanOcr::from_dir_with_dflash(
"tencent/HunyuanOCR",
parse_device("cuda:0")?,
)?;
assert!(model.dflash_enabled());
Ok(())
}
```
### Running the Example
```bash
cargo run -p oar-ocr-vl --features cuda --example hunyuanocr -- \
--model-dir tencent/HunyuanOCR \
--dflash-dir tencent/HunyuanOCR/dflash \
--device cuda \
--prompt "Detect and recognize text in the image, and output the text coordinates in a formatted manner." \
document.jpg
```
### Application-oriented Prompts
Prompts from the upstream HunyuanOCR README:
| Task | English | Chinese |
|------|---------|---------|
| **Spotting** | Detect and recognize text in the image, and output the text coordinates in a formatted manner. | 检测并识别图片中的文字，将文本坐标格式化输出。 |
| **Parsing** | • Identify the formula in the image and represent it using LaTeX format.<br><br>• Parse the table in the image into HTML.<br><br>• Parse the chart in the image; use Mermaid format for flowcharts and Markdown for other charts.<br><br>• Extract all information from the main body of the document image and represent it in markdown format, ignoring headers and footers. Tables should be expressed in HTML format, formulas in the document should be represented using LaTeX format, and the parsing should be organized according to the reading order. | • 识别图片中的公式，用 LaTeX 格式表示。<br><br>• 把图中的表格解析为 HTML。<br><br>• 解析图中的图表，对于流程图使用 Mermaid 格式表示，其他图表使用 Markdown 格式表示。<br><br>• 提取文档图片中正文的所有信息用 markdown 格式表示，其中页眉、页脚部分忽略，表格用 html 格式表达，文档中公式用 latex 格式表示，按照阅读顺序组织进行解析。 |
| **Information Extraction** | • Output the value of Key.<br><br>• Extract the content of the fields: ['key1','key2', ...] from the image and return it in JSON format.<br><br>• Extract the subtitles from the image. | • 输出 Key 的值。<br><br>• 提取图片中的: ['key1','key2', ...] 的字段内容，并按照 JSON 格式返回。<br><br>• 提取图片中的字幕。 |
| **Translation** | First extract the text, then translate the text content into English. If it is a document, ignore the header and footer. Formulas should be represented in LaTeX format, and tables should be represented in HTML format. | 先提取文字，再将文字内容翻译为英文。若是文档，则其中页眉、页脚忽略。公式用latex格式表示，表格用html格式表示。 |
## GLM-OCR
[GLM-OCR](https://huggingface.co/zai-org/GLM-OCR) is an OCR expert VLM in the `oar-ocr-vl` crate. It uses prompt-driven image-to-text generation and can be used directly or as a `DocParser` backend.
### Downloading the Model
```bash
git lfs install
git clone https://huggingface.co/zai-org/GLM-OCR
# Or using hf
hf download zai-org/GLM-OCR --local-dir GLM-OCR
```
### Basic Usage
```rust,no_run
use oar_ocr_vl::utils::image::load_image;
use oar_ocr_vl::GlmOcr;
use oar_ocr_vl::utils::parse_device;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let image = load_image("document.jpg")?;
let device = parse_device("cpu")?; // or "cuda", "cuda:0", "metal"
let model = GlmOcr::from_dir("zai-org/GLM-OCR", device)?;
let prompt = "Text Recognition:";
let text = model
.generate(&[image], &[prompt], 1024)?
.into_iter()
.next()
.expect("one result")?;
println!("{text}");
Ok(())
}
```
### Running the Example
```bash
cargo run -p oar-ocr-vl --features cuda --example glmocr -- \
--model-dir zai-org/GLM-OCR \
--device cuda \
--prompt "Text Recognition:" \
document.jpg
```
## MinerU2.5 and MinerU2.5-Pro
[MinerU2.5](https://huggingface.co/opendatalab/MinerU2.5-2509-1.2B) and [MinerU2.5-Pro](https://huggingface.co/opendatalab/MinerU2.5-Pro-2605-1.2B) are document parsing VLMs supported by the same `MinerU` loader. For full-page documents, use their model-native two-step pipeline rather than forcing them through `DocParser`.
### Downloading the Model
```bash
git lfs install
git clone https://huggingface.co/opendatalab/MinerU2.5-2509-1.2B
git clone https://huggingface.co/opendatalab/MinerU2.5-Pro-2605-1.2B
# Or using hf
hf download opendatalab/MinerU2.5-2509-1.2B --local-dir MinerU2.5-2509-1.2B
hf download opendatalab/MinerU2.5-Pro-2605-1.2B --local-dir MinerU2.5-Pro-2605-1.2B
```
### Basic Usage
```rust,no_run
use oar_ocr_vl::utils::image::load_image;
use oar_ocr_vl::{MinerU, MinerUParseOptions, PageParser};
use oar_ocr_vl::utils::parse_device;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let image = load_image("document.jpg")?;
let device = parse_device("cpu")?; // or "cuda", "cuda:0", "metal"
// MinerU2.5-Pro-2605-1.2B is loaded through the same API.
let model = MinerU::from_dir("opendatalab/MinerU2.5-2509-1.2B", device)?;
let document = model.parse_page(&image, &MinerUParseOptions::default())?;
println!("{:#?}", document.blocks);
Ok(())
}
```
### Running the Example
```bash
cargo run -p oar-ocr-vl --features cuda --example mineru -- \
--model-dir opendatalab/MinerU2.5-2509-1.2B \
--device cuda \
document.jpg
cargo run -p oar-ocr-vl --features cuda --example mineru -- \
--model-dir opendatalab/MinerU2.5-Pro-2605-1.2B \
--device cuda \
document.jpg
```
## MinerU-Diffusion-V1
[MinerU-Diffusion-V1](https://huggingface.co/opendatalab/MinerU-Diffusion-V1-0320-2.5B) replaces autoregressive text generation with block-diffusion decoding. The `mineru_diffusion` example defaults to MinerU-style two-step structured extraction and also provides `--single-pass` for flat full-page text recognition.
### Downloading the Model
```bash
git lfs install
git clone https://huggingface.co/opendatalab/MinerU-Diffusion-V1-0320-2.5B
# Or using hf
hf download opendatalab/MinerU-Diffusion-V1-0320-2.5B \
--local-dir MinerU-Diffusion-V1-0320-2.5B
```
### Basic Usage
```rust,no_run
use oar_ocr_vl::utils::image::load_image;
use oar_ocr_vl::mineru_diffusion::DEFAULT_PROMPT;
use oar_ocr_vl::utils::parse_device;
use oar_ocr_vl::{DiffusionGenerationConfig, MinerUDiffusion};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let image = load_image("document.jpg")?;
let model = MinerUDiffusion::from_dir(
"opendatalab/MinerU-Diffusion-V1-0320-2.5B",
parse_device("cuda:0")?,
)?;
let text = model.generate_one(
&image,
DEFAULT_PROMPT,
&DiffusionGenerationConfig::default(),
)?;
println!("{text}");
Ok(())
}
```
### Running the Example
```bash
cargo run -p oar-ocr-vl --features cuda \
--example mineru_diffusion -- \
--model-dir opendatalab/MinerU-Diffusion-V1-0320-2.5B \
--device cuda:0 \
document.jpg
```
## DocParser
DocParser provides a unified API for external layout-first document parsing with VL-based recognition. PaddleOCR-VL, GLM-OCR, MonkeyOCRv2, OvisOCR2, OvisOCR3, OXR, HunyuanOCR, MinerU2.5/Pro, and MinerU-Diffusion implement `RecognitionBackend`. The `doc_parser` CLI example exposes the PaddleOCR-VL and GLM-OCR paths; use the other backends through the library API. HPD-Parsing currently supports only its model-native full-page protocol.
Use `parse(&layout, image)` with any `LayoutSource` — `PpDocLayout` is the built-in one and runs on Candle, so no ONNX Runtime is involved. Regions are recognized in layout order and may be grouped into bounded recognition batches. For complete pages, prefer the model-native paths for MonkeyOCRv2, OvisOCR2, OvisOCR3, HPD-Parsing, HunyuanOCR, and the MinerU family. Except for HPD-Parsing, their `RecognitionBackend` implementations are intended for externally detected crops.
### Basic Usage
```rust
use oar_ocr_vl::utils::image::load_image;
use oar_ocr_vl::utils::parse_device;
use oar_ocr_vl::{DocParser, GlmOcr, PaddleOcrVl, PpDocLayout};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let device = parse_device("cpu")?;
// Initialize layout detector
let layout = PpDocLayout::from_dir("PaddlePaddle/PP-DocLayoutV3_safetensors", device.clone())?;
// Load document image
let image = load_image("document.jpg")?;
// Option 1: Using PaddleOCR-VL
let paddleocr_vl = PaddleOcrVl::from_dir("PaddlePaddle/PaddleOCR-VL", device.clone())?;
let parser = DocParser::new(&paddleocr_vl);
let result = parser.parse(&layout, image.clone())?;
println!("{}", result.to_markdown());
// Option 2: Using PaddleOCR-VL-1.5
let paddleocr_vl_15 =
PaddleOcrVl::from_dir("PaddlePaddle/PaddleOCR-VL-1.5", device.clone())?;
let parser = DocParser::new(&paddleocr_vl_15);
let result = parser.parse(&layout, image.clone())?;
println!("{}", result.to_markdown());
// Option 3: Using PaddleOCR-VL-1.6
let paddleocr_vl_16 =
PaddleOcrVl::from_dir("PaddlePaddle/PaddleOCR-VL-1.6", device.clone())?;
let parser = DocParser::new(&paddleocr_vl_16);
let result = parser.parse(&layout, image.clone())?;
println!("{}", result.to_markdown());
// Option 4: Using GLM-OCR with external layout
let glmocr = GlmOcr::from_dir("zai-org/GLM-OCR", device)?;
let parser = DocParser::new(&glmocr);
let result = parser.parse(&layout, image)?;
println!("{}", result.to_markdown());
Ok(())
}
```
### Running the Example
```bash
# Using PaddleOCR-VL
cargo run -p oar-ocr-vl --features cuda --example doc_parser -- \
--model-name paddleocr-vl \
--model-dir PaddlePaddle/PaddleOCR-VL \
--layout-dir PaddlePaddle/PP-DocLayoutV3_safetensors \
--device cuda \
document.jpg
# Using PaddleOCR-VL-1.5
cargo run -p oar-ocr-vl --features cuda --example doc_parser -- \
--model-name paddleocr-vl-1.5 \
--model-dir PaddlePaddle/PaddleOCR-VL-1.5 \
--layout-dir PaddlePaddle/PP-DocLayoutV3_safetensors \
--device cuda \
document.jpg
# Using PaddleOCR-VL-1.6
cargo run -p oar-ocr-vl --features cuda --example doc_parser -- \
--model-name paddleocr-vl-1.6 \
--model-dir PaddlePaddle/PaddleOCR-VL-1.6 \
--layout-dir PaddlePaddle/PP-DocLayoutV3_safetensors \
--device cuda \
document.jpg
# Using GLM-OCR with layout
cargo run -p oar-ocr-vl --features cuda --example doc_parser -- \
--model-name glmocr \
--model-dir zai-org/GLM-OCR \
--layout-dir PaddlePaddle/PP-DocLayoutV3_safetensors \
--device cuda \
document.jpg
```
## Configuration Options
### OrtSessionConfig
Control ONNX Runtime session behavior:
```rust
use oar_ocr::core::config::{OrtSessionConfig, OrtExecutionProvider};
let config = OrtSessionConfig::new()
.with_execution_providers(vec![OrtExecutionProvider::CPU])
.with_intra_threads(4)
.with_inter_threads(2);
```
Thread-count tuning is workload and CPU dependent. The OCR example exposes the relevant controls so Windows users can measure several values without changing code:
```powershell
cargo run --release --example ocr -- --intra-threads 8 <OPTIONS> <IMAGES>...
cargo run --release --example ocr -- --intra-threads 12 <OPTIONS> <IMAGES>...
cargo run --release --example ocr -- --intra-threads 16 <OPTIONS> <IMAGES>...
```
Pipelines with several ONNX models can share one process-wide worker pool instead of creating a pool for every session. Commit it before building any predictor:
```rust
use oar_ocr::core::OrtGlobalThreadPoolOptions;
OrtGlobalThreadPoolOptions::new()
.with_intra_threads(12)
.commit()?;
// Build OAROCR/OARStructure only after the global environment is committed.
```
The `ocr` and `structure` examples expose this as `--global-thread-pool`. Do not also set session-local thread counts when a global pool is active. Sharing mainly reduces worker creation and contention in multi-model pipelines; benchmark it for your workload rather than assuming it lowers single-request latency. See ONNX Runtime's [thread management guide](https://onnxruntime.ai/docs/performance/tune-performance/threading.html) for the underlying process-wide pool behavior.
On Windows, examples built with `--features directml` also accept `--device directml`, `--device directml:N`, and the shorter `--device dml:N`. The selected DirectML provider keeps the accelerator-oriented batch defaults.
### ONNX Runtime logging
OAR defaults ONNX Runtime's global logging to Error only when it initializes the environment itself; an environment configured by the application is preserved. Explicit session logging requests lower an OAR-owned environment's threshold to the lowest requested severity; subsequent sessions do not raise it again. To see ORT warnings, call `ort::init().commit()` and set `ort::environment::Environment::current()?.set_log_level(ort::logging::LogLevel::Warning)` before constructing any OAR model.
### Device-aware batching
The high-level OCR and Structure builders choose different defaults for CPU and accelerator execution. CPU operators already parallelize through ONNX Runtime's intra-op pool, so outer image batches default to `1` and text-recognition batches default to `4`, except PP-OCRv6 Tiny recognition uses `16` to better amortize its much smaller per-item workload. Explicitly configured accelerators retain the larger adapter defaults (`8` images for OCR detection, `4` pages for layout detection, and `64` text regions for recognition).
Explicit `.image_batch_size(...)` and `.region_batch_size(...)` values always take precedence. Model size and CPU topology still matter, so applications with fixed workloads should benchmark nearby values.
For a 4 GiB GPU, add `.gpu_memory_budget(4 * 1024 * 1024 * 1024)` to either OCR or Structure builder after selecting an accelerator. With whole GiB `g`, image batches use `clamp(2*g - 6, 1, 8)` and recognition uses `clamp(12*g - 28, 4, 64)`: 4 GiB gives 2/20, and 8 GiB or more gives 8/64. Explicit batch sizes still win. CUDA arenas are limited to half the budget per session, default to `SameAsRequested`, and release idle memory; existing smaller arena limits and explicit arena settings are retained. The budget is a tuning hint, not measured free VRAM or a device-wide memory cap: resident models, concurrent pipelines, and non-arena allocations can exceed it. DirectML gets the conservative batches but cannot use CUDA's arena controls.
### Task-Specific Configs
Each task has its own configuration struct that can be customized:
```rust
use oar_ocr::domain::TextDetectionConfig;
let det_config = TextDetectionConfig {
score_threshold: 0.3,
box_threshold: 0.6,
unclip_ratio: 1.5,
max_candidates: 1000,
..Default::default()
};
```