stygian-plugin 0.14.1

Visual data extraction fallback subsystem with CSS/XPath selectors, idempotent request handling, and composable transformation pipelines.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147

//! stygian-plugin: Chrome browser plugin fallback scraper
//!
//! Provides a flexible, interactive visual data extraction framework as a fallback
//! when stygian-graph and stygian-browser cannot scrape a page.
//!
//! # Architecture
//!
//! Following hexagonal architecture with clear separation:
//!
//! ```text
//! ┌─────────────────────────────────────┐
//! │  Application / MCP Layer            │
//! │  (plugin_apply_template, etc.)      │
//! └──────────────┬──────────────────────┘
//!                │
//! ┌──────────────▼──────────────────────┐
//! │  Domain Layer (pure Rust)           │
//! │  ExtractionTemplate                 │
//! │  ExtractionRequest/Result           │
//! │  Transformation Pipeline            │
//! └──────────────┬──────────────────────┘
//!                │
//! ┌──────────────▼──────────────────────┐
//! │  Ports (traits)                     │
//! │  PluginTemplateStore                │
//! │  PluginExtractionPort               │
//! │  IdempotencyKeyStore                │
//! └──────────────┬──────────────────────┘
//!                │
//! ┌──────────────▼──────────────────────┐
//! │  Adapters (implementations)         │
//! │  FileTemplateStore                  │
//! │  ExtractionEngine                   │
//! │  MemoryIdempotencyStore             │
//! └─────────────────────────────────────┘
//! ```
//!
//! # Features
//!
//! - **Template-based extraction**: Define schema once, apply to multiple elements
//! - **Recording-based**: User clicks/highlights → learns pattern
//! - **Query-driven**: Declarative extraction with CSS/XPath selectors
//! - **Region-based**: Multiple independent zones, each with own rules
//! - **Multi-instance**: Iterate template across matching elements
//! - **Multi-set**: Extract different shapes from same page
//! - **Cross-page**: Reuse templates in crawl sessions
//! - **Idempotency**: Safe retries via ULID-based deduplication
//! - **Transformations**: Regex, type coercion, HTML stripping, etc.
//!
//! # Quick Start
//!
//! ```no_run
//! use stygian_plugin::domain::{ExtractionTemplate, Region, Selector, ExtractionRequest};
//! use stygian_plugin::ports::PluginExtractionPort;
//! use serde_json::json;
//!
//! # async fn example() -> Result<(), Box<dyn std::error::Error>> {
//! // Create a template with regions
//! let template = ExtractionTemplate::new("Product")
//!     .with_region(
//!         Region::new(
//!             "title",
//!             Selector::css(".product-title"),
//!             json!({"type": "string"}),
//!         )
//!     )
//!     .with_region(
//!         Region::new(
//!             "price",
//!             Selector::css(".product-price"),
//!             json!({"type": "number"}),
//!         )
//!     );
//!
//! // Create extraction request
//! let request = ExtractionRequest::new(
//!     template,
//!     "https://example.com/products",
//!     "<html>...</html>"
//! );
//!
//! // Execute (requires a PluginExtractionPort adapter)
//! // let result = extraction_port.execute(&request).await?;
//! # Ok(())
//! # }
//! ```

#![allow(clippy::multiple_crate_versions)]

// ═══════════════════════════════════════════════════════════════════════════
// Module Organization
// ═══════════════════════════════════════════════════════════════════════════

/// Error types
pub mod error;

/// Domain layer: pure business logic and value objects
///
/// Contains zero external dependencies; all I/O happens in adapters.
pub mod domain;

/// Port trait definitions: interfaces adapters must implement
///
/// The domain depends only on these traits, not on concrete implementations.
pub mod ports;

/// Adapter implementations: concrete providers of port traits
pub mod adapters;

/// Storage adapters: template persistence, idempotency tracking
pub mod storage;

/// MCP (Model Context Protocol) server for the plugin system
pub mod mcp;

/// HTTP transport for the MCP server (requires `http` feature)
///
/// Exposes JSON-RPC 2.0 over HTTP with CORS support for browser extension use.
#[cfg(feature = "http")]
pub mod http;

/// Runtime configuration for the standalone MCP server
pub mod config;

/// Extraction reliability scoring
///
/// Computes a 0.0–1.0 reliability score for [`domain::ExtractionResult`]
/// outputs so fallback chains can optimize for *data quality*, not only
/// fetch success. See the module-level docs for the score interpretation
/// table and the selection policy.
pub mod reliability;

// ═══════════════════════════════════════════════════════════════════════════
// Public API Re-exports
// ═══════════════════════════════════════════════════════════════════════════

pub use domain::{
    ExtractionRequest, ExtractionResult, ExtractionTemplate, IdempotencyKey, Region, Selector,
    Transformation,
};
pub use error::{PluginError, Result};
pub use mcp::{McpPluginServer, McpRequestHandler};
pub use ports::{IdempotencyKeyStore, PluginExtractionPort, PluginTemplateStore};
pub use reliability::{
    ReliabilityBand, ReliabilityScore, ReliabilityScorer, ScoreWeightedSelector, ScoredCandidate,
    ScoringWeights,
};