Struct Storage 

pub struct Storage { /* private fields */ }

SQLite-backed storage for crawl data.

Uses WAL mode for concurrent-safe access and provides batch-friendly insert methods for performance. Includes an LRU cache for frequently accessed pages and memory usage tracking.

Implementations

impl Storage

pub fn conn(&self) -> MutexGuard<'_, Connection>

Lock and return the underlying connection guard.

Provides direct access to the SQLite connection for advanced queries or transactions not covered by the convenience methods.

pub fn new(path: &Path) -> Result<Self, StorageError>

Open or create a SQLite database at the given path.

Enables WAL mode, memory-mapped I/O, and creates the schema if it doesn’t exist. Uses a default LRU cache of 1000 entries.

Errors

Returns StorageError::Database if the database cannot be opened or the schema cannot be created.

pub fn new_in_memory() -> Result<Self, StorageError>

Create in-memory storage for testing.

Uses an in-memory SQLite database with WAL mode enabled. Ideal for unit tests that need fast, isolated storage.

Errors

Returns StorageError::Database if the in-memory database cannot be created.

pub fn with_cache_size( path: &Path, cache_size: usize, ) -> Result<Self, StorageError>

Create storage with a custom LRU cache size.

Use this when the default 1000-entry cache is not appropriate for your workload. Larger caches improve read performance for repeat queries but use more memory.

Errors

Returns StorageError::Database if the database cannot be opened.

pub fn memory_usage(&self) -> usize

Returns the approximate memory usage in bytes.

pub fn is_mmap_enabled(&self) -> bool

Returns whether memory-mapped I/O is enabled.

pub fn cache_stats(&self) -> CacheStats

Returns cache statistics (hits, misses, current size).

pub fn clear_cache(&self)

Clears the page cache.

pub fn start_crawl( &self, target_url: &str, config_json: Option<&str>, ) -> Result<String, StorageError>

Start a new crawl and return its ID.

pub fn finish_crawl( &self, crawl_id: &str, pages_crawled: usize, total_issues: usize, ) -> Result<(), StorageError>

Finish a crawl, recording final statistics.

pub fn insert_page( &self, crawl_id: &str, page: &PageData, ) -> Result<(), StorageError>

Insert a single page into the database under the given crawl. Uses a single SQLite transaction for the page row + all link rows to avoid per-statement fsync overhead.

pub fn insert_pages( &self, crawl_id: &str, pages: &[PageData], ) -> Result<(), StorageError>

Insert a batch of pages for performance. Wraps all inserts in a single SQLite transaction for O(n) vs O(n*fsync).

pub fn insert_issue(&self, issue: &Issue) -> Result<(), StorageError>

Insert a single issue/finding into the database.

pub fn insert_issues(&self, issues: &[Issue]) -> Result<(), StorageError>

Insert a batch of issues for performance.

pub fn get_pages( &self, crawl_id: &str, limit: usize, ) -> Result<Vec<PageData>, StorageError>

Retrieve pages with a limit.

Results are not cached because the cache type (LruCache<String, PageData>) cannot store Vec<PageData>. The query is fast with proper indexing.

pub fn get_issues( &self, crawl_id: &str, filters: &IssueFilter, ) -> Result<Vec<Issue>, StorageError>

Retrieve issues/finding with optional filters.

pub fn get_pages_for_tenant( &self, crawl_id: &str, tenant_id: &str, limit: usize, ) -> Result<Vec<PageData>, StorageError>

Get pages for a specific tenant.

Returns pages belonging to the given tenant, or pages with no tenant assigned (shared/global data). This ensures tenant isolation at the storage layer while still allowing access to unscoped data.

pub fn get_issues_for_tenant( &self, crawl_id: &str, tenant_id: &str, filters: &IssueFilter, ) -> Result<Vec<Issue>, StorageError>

Get issues for a specific tenant.

Returns issues belonging to the given tenant, or issues with no tenant assigned (shared/global data). This ensures tenant isolation at the storage layer while still allowing access to unscoped data.

pub fn get_stats(&self, crawl_id: &str) -> Result<CrawlStats, StorageError>

Get aggregate statistics for a crawl.

pub fn get_latest_crawl_id(&self) -> Result<Option<String>, StorageError>

Get the latest crawl ID.

pub fn get_links_for_crawl( &self, crawl_id: &str, ) -> Result<Vec<(String, Vec<String>)>, StorageError>

Get all links for a crawl, grouped by source URL.

Returns Vec<(source_url, Vec<target_url>)> suitable for feeding into BacklinkAnalyzer::load_from_crawl_data.

pub fn get_external_links( &self, crawl_id: &str, ) -> Result<Vec<(String, String)>, StorageError>

Get all external links for a crawl.

pub fn get_page_urls(&self, crawl_id: &str) -> Result<Vec<String>, StorageError>

Get all page URLs for a crawl.

pub fn insert_crux_metrics( &self, page_id: &str, url: &str, lcp_p75: Option<f64>, inp_p75: Option<f64>, cls_p75: Option<f64>, fcp_p75: Option<f64>, ttfb_p75: Option<f64>, ) -> Result<(), StorageError>

Store CrUX metrics for a page.

pub fn get_crux_metrics( &self, page_id: &str, ) -> Result<Option<CruxMetrics>, StorageError>

Get CrUX metrics for a page.

pub fn get_crux_metrics_for_crawl( &self, crawl_id: &str, ) -> Result<Vec<CruxMetrics>, StorageError>

Get CrUX metrics for all pages in a crawl.