pub struct Response {
pub url: Url,
pub status: StatusCode,
pub headers: HeaderMap,
pub body: Bytes,
pub request_url: Url,
pub meta: Option<Arc<DashMap<String, Value>>>,
pub cached: bool,
}Expand description
Represents an HTTP response received from a server.
Response contains all information about an HTTP response, including
the final URL (after redirects), status code, headers, body content,
and metadata carried over from the original request.
The type is designed for parse-time ergonomics:
Response::to_htmlparses the body as HTMLResponse::jsondeserializes JSON payloadsResponse::linksand related helpers extract follow-up links- [
Response::to_request] reconstructs the originating request context
§Example
use spider_util::response::Response;
use reqwest::StatusCode;
use bytes::Bytes;
use url::Url;
let response = Response {
url: Url::parse("https://example.com").unwrap(),
status: StatusCode::OK,
headers: http::header::HeaderMap::new(),
body: Bytes::from("<html><body>Hello</body></html>"),
request_url: Url::parse("https://example.com").unwrap(),
meta: None,
cached: false,
};
// Parse the response body as HTML
if let Ok(html) = response.to_html() {
// Process HTML...
}Fields§
§url: UrlThe final URL of the response after any redirects.
status: StatusCodeThe HTTP status code of the response.
headers: HeaderMapThe headers of the response.
body: BytesThe body of the response.
request_url: UrlThe original URL of the request that led to this response.
meta: Option<Arc<DashMap<String, Value>>>Metadata associated with the response, carried over from the request. Uses Option to allow lazy initialization.
cached: boolIndicates if the response was served from a cache.
Implementations§
Source§impl Response
impl Response
Sourcepub fn new(
url: Url,
status: StatusCode,
headers: HeaderMap,
body: Bytes,
request_url: Url,
) -> Self
pub fn new( url: Url, status: StatusCode, headers: HeaderMap, body: Bytes, request_url: Url, ) -> Self
Creates a new response with an empty HTML cache.
Most application code receives responses from the runtime rather than constructing them directly. This constructor is mainly useful for custom downloaders and lower-level integrations.
Sourcepub fn request_from_response(&self) -> Request
pub fn request_from_response(&self) -> Request
Reconstructs the original Request that led to this response.
This method creates a new Request with the same URL and metadata
as the request that produced this response. Useful for retry scenarios
or when you need to re-request the same resource.
§Example
let original_request = response.request_from_response();Sourcepub fn json<T: DeserializeOwned>(&self) -> Result<T, Error>
pub fn json<T: DeserializeOwned>(&self) -> Result<T, Error>
Deserializes the response body as JSON.
§Type Parameters
T: The target type to deserialize into (must implementDeserializeOwned)
§Errors
Returns a serde_json::Error if the body cannot be parsed as JSON
or if it cannot be deserialized into type T.
§Example
let data: Data = response.json()?;Sourcepub fn to_html(&self) -> Result<Html, Utf8Error>
pub fn to_html(&self) -> Result<Html, Utf8Error>
Parses the response body as HTML.
Returns a scraper::Html document that can be queried using CSS selectors.
§Errors
Returns a Utf8Error if the response body is not valid UTF-8.
§Example
let html = response.to_html()?;Sourcepub fn lazy_html(
&self,
) -> Result<impl Fn() -> Result<Html, Utf8Error> + '_, Utf8Error>
pub fn lazy_html( &self, ) -> Result<impl Fn() -> Result<Html, Utf8Error> + '_, Utf8Error>
Lazily parses the response body as HTML.
Returns a closure that can be called when the HTML is actually needed. This avoids parsing HTML for responses where it may not be used.
§Errors
Returns a Utf8Error if the response body is not valid UTF-8.
§Example
let html_fn = response.lazy_html()?;
// Parse HTML only when needed
let html = html_fn()?;Sourcepub fn links_iter(
&self,
options: LinkExtractOptions,
) -> impl Iterator<Item = Link>
pub fn links_iter( &self, options: LinkExtractOptions, ) -> impl Iterator<Item = Link>
Returns a customizable iterator of links discovered in the response body.
Unlike Response::links, this method does not deduplicate results.
Callers that need uniqueness can collect into a set or use Response::links.
§Example
let links: Vec<_> = response
.links_iter(LinkExtractOptions::default())
.collect();
assert!(!links.is_empty());Sourcepub fn links(&self) -> DashSet<Link>
pub fn links(&self) -> DashSet<Link>
Extracts all unique, same-site links from the response body.
This method discovers links from:
- HTML elements with
hreforsrcattributes (<a>,<link>,<script>,<img>, etc.) - URLs found in text content (using link detection)
Only links pointing to the same site (same registered domain) are included.
§Returns
A DashSet of Link objects containing the URL and link type.
§Example
let links = response.links();
for link in links.iter() {
println!("Found {:?} link: {}", link.link_type, link.url);
}