Module referer 

Referer Middleware for managing HTTP Referer headers.

This module provides the RefererMiddleware, which automatically sets the Referer HTTP header for outgoing requests. Its primary purpose is to simulate natural browsing behavior, making web crawls appear more legitimate and potentially bypassing certain anti-scraping measures.

Key features include:

• Automatic tracking of the request chain to determine the appropriate referer.
• Configuration options for enforcing same-origin referers, limiting the referer chain length, and controlling the inclusion of URL fragments.
• Dynamic insertion of the Referer header into outgoing requests.

Structs

RefererMiddleware

Referer middleware that automatically sets Referer headers based on the navigation chain