Crate knee_scraper

source

Functionsยง

  • check_open_directories
    Checks for common open directories on the server.
  • download_media
    Downloads a media file (image or video) and saves it to the local directory.
  • extract_domain
    Extracts the domain from a URL for folder naming purposes.
  • extract_links
    Extracts all links from an HTML page, normalizing them to absolute URLs.
  • fetch_robots_txt
    Fetches and parses the robots.txt file.
  • fetch_with_cookies
    Fetches a web page and prints the response status, demonstrating cookie handling.
  • normalize_link
    Normalizes a link to an absolute URL based on the base URL.
  • random_delay
    Sleeps for a random duration between a given range, mimicking human browsing behavior.
  • random_user_agent
    Generates a random user-agent string from a predefined list.
  • recursive_scrape
  • run
    Executes the entire scraping workflow for the provided URL, including:
  • scrape_content
    Scrapes all meaningful content from an HTML page, including text, images, videos, meta tags, and forms.
  • scrape_for_emails
    Scrapes for emails and saves them to a file.
  • scrape_for_errors
    Scrapes for errors and stack traces in the HTML content.
  • scrape_js_content
    Scrapes JavaScript content for API keys or tokens.