# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [0.2.2] - 2026-08-05
Upstream v0.4.12 sync (AutoThrottle, CSV/XML export) plus the results of
a full internal audit: seven silent-corruption bug fixes, engine
concurrency fixes, complete robots.txt matching, `::text`/`::attr()`
support, and a complete rustdoc pass.
### Added
- **AutoThrottle** (upstream v0.4.12): adaptive per-domain crawl delays
driven by measured response latency — 429/503 double the delay or
honor `Retry-After`, healthy responses ease it back down;
`download_delay` and robots.txt `Crawl-delay` are floors. Opt-in via
`Spider::autothrottle_enabled()`; spacing uses a per-domain
reservation clock so domains never delay each other
- **CSV and XML export** (upstream v0.4.12): `ItemList::to_csv` (RFC
4180, header = union of keys across heterogeneous items, nested
values as JSON strings) and `ItemList::to_xml` (entity-escaped,
XML-invalid control characters stripped, element names sanitized)
- **`::text` and `::attr(name)` pseudo-elements**: Parsel-inspired
`css_get`/`css_getall` on `Selector` and `SpiderResponse`
- **Complete robots.txt matching**: `Allow:` directives with
longest-match-wins evaluation (Allow wins ties), `*`/`$` wildcards
matched against path + query, rules scoped per scheme + authority
(RFC 9309), combined same-agent groups, user-agent product-token
matching, robots.txt fetched from the request's actual origin
(scheme/host/port) instead of hardcoded `https://host/`
- Per-domain (`domains_response_bytes`) and per-session
(`sessions_requests_count`) crawl statistics are now populated
- Complete rustdoc documentation: crate overview with examples, every
public item documented (`missing_docs` enforced), `#[must_use]` on
builders and query methods
### Changed
- **Engine concurrency**: the per-domain semaphore is acquired inside
the spawned task instead of the dispatch loop, so one saturated
domain no longer stalls dispatch for every other domain; the
robots.txt network fetch no longer holds the manager lock (up to 10s)
while other tasks wait to check `is_allowed`; dev-cache replays skip
the download delay entirely
- `serde_json` now uses `preserve_order`: item keys keep insertion
order (Python dict parity), so exports show fields in the order
spiders build them. Note for consumers: Cargo feature unification
enables insertion-ordered `serde_json::Map` crate-wide; and
fingerprints of requests whose `meta` contains nested objects can
differ from ones persisted by 0.2.1 (a resumed old checkpoint may
re-fetch such URLs once)
- `tokio` dependency trimmed from `full` to the features actually used;
`once_cell` replaced by `std::sync::LazyLock`; unused `http` and
`tokio-test` dependencies removed
### Fixed
- **Per-request headers were never sent**: headers set via
`SpiderRequestBuilder::header()` now reach the wire
- **Checkpoints preserved only URLs**: pending requests now persist in
full (method, headers, body, meta, priority); old URL-only
checkpoints still restore
- **Blocked responses were cached in dev mode**, so retries replayed
the same blocked page from disk forever; the `is_blocked` check now
runs before the cache write
- HTML entity decoding no longer double-decodes escaped entities
(`&lt;` now correctly yields the literal text `<`)
- `find_by_text` matched different text depending on `first_match`;
both paths now use recursive text with deepest-match-first descent
- `SpiderRequest`'s `Ord`/`PartialEq` contract violation fixed (tie
break on fingerprint)
- `SqliteStorage::retrieve` no longer swallows real database errors as
"not found"
- `ProxyRotator::random()` now advances its cursor
- Hostile robots.txt `Crawl-delay` values (`inf`, `NaN`, huge numbers)
can no longer panic request tasks
## [0.2.1] - 2026-07-17
First release shipped through the automated release pipeline (tag,
GitHub release, and crates.io publish are created by CI on version bump).
### Added
- `documentation` link (docs.rs) in the crate metadata, shown on the crates.io page
## [0.2.0] - 2026-07-17
First release published to crates.io. Syncs all applicable changes from
upstream Scrapling v0.4.8 through v0.4.11.
### Added
- **Adaptive element relocation**: `Selector::save`/`retrieve`/`relocate` and `css_adaptive` — persist an element's fingerprint to SQLite and find it again by similarity after the page structure changes. Difflib-parity sequence matcher (position index + autojunk), default threshold 40%
- **`LinkExtractor`**: declarative URL discovery with regex allow/deny, domain filters (subdomain-aware), `restrict_css` scoping, URL canonicalization, and a binary-extension deny list including compound extensions (`.tar.gz`)
- **`CrawlSpider` + `CrawlRule`**: follow links matching declarative rules, with priority override and `process_request` hook
- **`SitemapSpider`**: seed crawls from sitemaps or robots.txt `Sitemap:` directives, recurse through sitemap indexes (filtered by `sitemap_follow`), dispatch URLs through rules
- **`ShopifySpider`**: extract every product variant from any Shopify store via its JSON API (`collections.json` → `products.json`), no HTML parsing (upstream v0.4.11)
- **Benchmarks**: reproducible parse+extract comparison vs Python Scrapling (`scripts/benchmark/`, `examples/benchmark.rs`)
### Changed
- **Charset-aware response decoding**: bodies are decoded per the `Content-Type` charset via `encoding_rs` (quoted values, whitespace tolerance, replacement-encoding fallback) instead of always lossy UTF-8
- **Checkpoints persist the dedup set**: resumed crawls no longer re-visit URLs crawled before a pause; checkpoints are written compactly
- README rewritten: animated hero, plain-language intro, benchmark results, AI-agent integration patterns
### Fixed
- `LinkExtractor` extracts in true document order across tags; empty queries normalize away; non-UTF-8 percent-escapes survive canonicalization
- Sitemap parsing is robust to html5ever's re-nesting of self-closing elements; relative sitemap URLs resolve against the response URL; content pages mentioning `<urlset` in scripts are no longer misclassified
## [0.1.0] - 2026-05-04
### Added
- **Core types**: `TextHandler` (string wrapper with regex, JSON, cleaning), `TextHandlers` (batch ops), `AttributesHandler` (read-only attribute map)
- **SQLite storage**: Persistent element tracking for adaptive mode via `SqliteStorage`
- **HTML parser**: `Selector` with CSS selector support, text extraction, DOM navigation (parent, children, siblings, next, previous)
- **Selector generation**: Auto-generate CSS/XPath selectors from element position
- **CSS translator**: Support for `::text` and `::attr()` pseudo-elements
- **HTTP client**: Async `Fetcher` with configurable retries, stealth headers, proxy support
- **Fetcher config**: Builder pattern with timeout, retries, proxy, user agent, stealth headers
- **Proxy rotation**: Round-robin and random proxy selection
- **Response integration**: `Response` auto-parses to `Selector` for immediate HTML querying
- **Spider trait**: User-facing API with configurable concurrency, rate limiting, domain filtering
- **Crawler engine**: Async orchestrator with `tokio`, semaphore-based concurrency, robots.txt compliance
- **Request fingerprinting**: SHA-256 based deduplication in priority scheduler
- **Dev-mode caching**: Disk-based response cache for development iteration
- **Checkpoint system**: Pause/resume support for long-running crawls
- **robots.txt**: Automatic compliance with Disallow rules and Crawl-delay
- **CLI**: `fetch` and `extract` subcommands with CSS selector and format options
- **175 tests** covering all modules
[0.2.2]: https://github.com/Liohtml/RUSTScrapling/releases/tag/v0.2.2
[0.2.1]: https://github.com/Liohtml/RUSTScrapling/releases/tag/v0.2.1
[0.2.0]: https://github.com/Liohtml/RUSTScrapling/releases/tag/v0.2.0
[0.1.0]: https://github.com/Liohtml/RUSTScrapling/releases/tag/v0.1.0