Module spider::utils

Application utils.

Modules

• header_utils
  Utils to modify the HTTP header.

Structs

• HttpResponse
  A basic generic type that represents an HTTP response.
• PageResponse
  The response of a web page.

Enums

• HttpVersion
  Represents an HTTP version

Functions

• clean_html
  Clean the html removing css and js
• clean_html_raw
  Clean the html removing css and js default using the scraper crate.
• clean_html_slim
  Clean and remove all base64 images from the prompt.
• fetch_page_html
  Perform a network request to a resource extracting all content as text streaming.
• fetch_page_html_raw
  Perform a network request to a resource extracting all content streaming.
• handle_openai_credits
  Handle the OpenAI credits used. This does nothing without ‘openai’ feature flag.
• log
  Log to console if configuration verbose.
• openai_request
  Perform a request to OpenAI Chat. This does nothing without the ‘openai’ flag enabled.
• put_hybrid_cache
  Store the page to cache to be re-used across HTTP request.