susbot 2.0.1

robots.txt checker for the command line and CI (sus.bot). Installs the same `susbot` program as the susbot-cli crate.
susbot-2.0.1 is not a library.

susbot

The command-line version of sus.bot, a robots.txt checker: which search engines and AI crawlers a site lets in, what is wrong with the file, and which paths it reveals.

cargo install susbot
susbot https://example.com

This crate installs the same susbot program as susbot-cli, where the usage is documented. Source and GitHub Action: github.com/sitefig/robots.

Licence

PolyForm Noncommercial 1.0.0: free for noncommercial use. Commercial use needs a licence from Sitefig.