Struct FetchStreamReader 

pub struct FetchStreamReader {
    pub final_url: Url,
    pub status_code: u16,
    pub headers: Vec<(String, String)>,
    pub response_time: Duration,
    pub body_size: usize,
    /* private fields */
}

A streaming HTTP response reader.

Read chunks from the body using the next_chunk method. The stream automatically respects max_body_size. Can be converted into a FetchResult via into_fetch_result.

Examples

use crawlkit_engine::{CrawlConfig, HttpClient};
use url::Url;

let client = HttpClient::from_crawl_config(&CrawlConfig::default())?;
let url = Url::parse("https://example.com")?;
let mut reader = client.fetch_reader(&url).await?;
while let Some(chunk) = reader.next_chunk().await? {
    // process chunk
}

Fields

final_url: Url

status_code: u16

headers: Vec<(String, String)>

response_time: Duration

body_size: usize

Implementations

impl FetchStreamReader

pub async fn next_chunk(&mut self) -> Result<Option<Vec<u8>>, CrawlError>

Reads the next chunk of the response body.

Returns Ok(Some(bytes)) if data is available, Ok(None) if the stream is complete, or Err on error. Respects max_body_size.

Errors

Returns CrawlError::RequestFailed on network errors.

pub async fn read_body(&mut self) -> Result<String, CrawlError>

Reads the entire remaining body into a String.

Convenience method that drains all remaining chunks and concatenates them into a single UTF-8 string (lossy conversion).

Errors

Returns CrawlError::RequestFailed on network errors.

pub async fn into_fetch_result(self) -> Result<FetchResult, CrawlError>

Converts this into a FetchResult by reading the full body.

Consumes the reader and returns the complete response including headers, status, and body content.

Errors

Returns CrawlError::RequestFailed on network errors.