# Changelog
## 0.1.19
- Add a frontier status/depth/url SQLite index to speed pending queue selection during long crawls.
## 0.1.18
- Batch in-progress frontier marking with a reused SQLite statement to reduce crawl-loop database overhead.
## 0.1.17
- Cache SQLite FTS availability at storage open time so page indexing and search avoid repeated schema checks.
## 0.1.16
- Preallocate streamed response buffers from valid `Content-Length` hints to reduce reallocations on large allowed page and asset downloads.
## 0.1.15
- Avoid repeated crawl stat aggregation for each job snapshot row in dashboard/diagnostic views.
- Treat rapid fetching/processing progress flips as hot-path updates so long crawls avoid unnecessary SQLite counter refreshes.
## 0.1.14
- De-duplicate concurrent `robots.txt` fetches per origin so high-concurrency same-site crawls do not all block on duplicate robots requests.
- Throttle progress stat refreshes to phase changes and periodic updates, reducing repeated SQLite counter scans during long crawls.
- Whitelist Cargo package contents to source, tests, config example, README, and changelog.