SuckIT
SuckIT allows you to recursively visit and download a website's content to
your disk.
Features
- Vacuums the entirety of a website recursively
- Uses multithreading
- Writes the website's content to your disk
- Enables offline navigation
- Offers random delays to avoid IP banning
- Saves application state on CTRL-C for later pickup
Options
| Option | Behavior |
|---|---|
-h, --help |
Displays help information |
-v, --verbose |
Activate Verbose output |
-d, --depth |
Specify the level of depth to go to when visiting the website |
-j, --jobs |
Number of threads to use |
-o, --output |
Output directory where the downloaded files are written |
-t, --tries |
Number of times to retry when the downloading of a page fails |
-u, --user-agent |
User agent to be used for sending requests |
-i, --include |
Specify a regex to include pages that match this pattern |
-e, --exclude |
Specify a regex to exclude pages that match this pattern |
-a, --auth |
Provide usernames and passwords for the downloader to use |
--dry-run |
Do everything without saving the files to the disk |
Example
A common use case could be the following:
suckit http://books.toscrape.com -j 8 -o /path/to/downloaded/pages/
Installation
As of right now, SuckIT does not work on Windows.
To install it, you need to have Rust installed.
-
Check out this link for instructions on how to install Rust.
-
If you just want to install the suckit executable, you can simply run
cargo install --git https://github.com/skallwar/suckit -
Now, run it from anywhere with the
suckitcommand.
Arch Linux
suckit can be installed from available AUR packages using an AUR helper. For example,
yay -S suckit
Want to contribute ? Feel free to open an issue or submit a PR !
License
SuckIT is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0)
See LICENSE-APACHE and LICENSE-MIT for details.