pub struct Response {
pub url: Url,
pub status: StatusCode,
pub headers: HeaderMap,
pub body: Bytes,
pub request_url: Url,
pub meta: Option<Arc<DashMap<String, Value>>>,
pub cached: bool,
}Expand description
Represents an HTTP response received from a server.
Response contains all information about an HTTP response, including
the final URL (after redirects), status code, headers, body content,
and metadata carried over from the original request.
§Example
use spider_util::response::Response;
use reqwest::StatusCode;
use bytes::Bytes;
use url::Url;
let response = Response {
url: Url::parse("https://example.com").unwrap(),
status: StatusCode::OK,
headers: http::header::HeaderMap::new(),
body: Bytes::from("<html><body>Hello</body></html>"),
request_url: Url::parse("https://example.com").unwrap(),
meta: None,
cached: false,
};
// Parse the response body as HTML
if let Ok(html) = response.to_html() {
// Process HTML...
}Fields§
§url: UrlThe final URL of the response after any redirects.
status: StatusCodeThe HTTP status code of the response.
headers: HeaderMapThe headers of the response.
body: BytesThe body of the response.
request_url: UrlThe original URL of the request that led to this response.
meta: Option<Arc<DashMap<String, Value>>>Metadata associated with the response, carried over from the request. Uses Option to allow lazy initialization.
cached: boolIndicates if the response was served from a cache.
Implementations§
Source§impl Response
impl Response
Sourcepub fn request_from_response(&self) -> Request
pub fn request_from_response(&self) -> Request
Reconstructs the original Request that led to this response.
This method creates a new Request with the same URL and metadata
as the request that produced this response. Useful for retry scenarios
or when you need to re-request the same resource.
§Example
let original_request = response.request_from_response();Sourcepub fn json<T>(&self) -> Result<T, Error>where
T: DeserializeOwned,
pub fn json<T>(&self) -> Result<T, Error>where
T: DeserializeOwned,
Deserializes the response body as JSON.
§Type Parameters
T: The target type to deserialize into (must implementDeserializeOwned)
§Errors
Returns a serde_json::Error if the body cannot be parsed as JSON
or if it cannot be deserialized into type T.
§Example
let data: Data = response.json()?;Sourcepub fn to_html(&self) -> Result<Html, Utf8Error>
pub fn to_html(&self) -> Result<Html, Utf8Error>
Parses the response body as HTML.
Returns a scraper::Html document that can be queried using CSS selectors.
§Errors
Returns a Utf8Error if the response body is not valid UTF-8.
§Example
let html = response.to_html()?;Sourcepub fn lazy_html(&self) -> Result<impl Fn(), Utf8Error>
pub fn lazy_html(&self) -> Result<impl Fn(), Utf8Error>
Lazily parses the response body as HTML.
Returns a closure that can be called when the HTML is actually needed. This avoids parsing HTML for responses where it may not be used.
§Errors
Returns a Utf8Error if the response body is not valid UTF-8.
§Example
let html_fn = response.lazy_html()?;
// Parse HTML only when needed
let html = html_fn()?;Sourcepub fn links(&self) -> DashSet<Link>
pub fn links(&self) -> DashSet<Link>
Extracts all unique, same-site links from the response body.
This method discovers links from:
- HTML elements with
hreforsrcattributes (<a>,<link>,<script>,<img>, etc.) - URLs found in text content (using link detection)
Only links pointing to the same site (same registered domain) are included.
§Returns
A DashSet of Link objects containing the URL and link type.
§Example
let links = response.links();
for link in links.iter() {
println!("Found {:?} link: {}", link.link_type, link.url);
}