pandubbi 0.1.2

A high-performance Rust web crawler engine developed by Shiv Studio.
pandubbi-0.1.2 is not a library.

🌊 Pandubbi (पनडुब्बी)

A lightweight, asynchronous, high-performance web crawler engine written in Rust.

Developed and maintained by SHIV STUDIO.

🚀 Features

  • Async Engine: Built with tokio and reqwest for fast concurrent crawling.
  • HTML Parsing: Uses scraper for efficient link and data extraction.
  • URL Handling: Safe URL normalization with url.

🛠️ Requirements

  • Rust (Edition 2021)
  • Cargo

💻 Quick Start

# Clone the repository
git clone [https://github.com/hidegammer867-a11y/pandubbi.git](https://github.com/hidegammer867-a11y/pandubbi.git)

# Navigate to project directory
cd pandubbi

# Run the crawler
cargo run