rss-miner
A CLI tool that finds RSS feeds from URLs and generates a valid OPML file.
Features
- Parallel Processing: Uses Rayon to process multiple URLs concurrently
- RSS Feed Validation: Validates RSS/Atom feeds before including them
- OPML Generation: Creates a valid OPML file compatible with feed readers
- Auto-Discovery: Finds RSS feeds in HTML link tags and common feed paths
- Error Handling: Robust error handling with detailed feedback
Installation
Usage
Arguments
-i, --input <FILE>: Input file containing URLs (one per line, required)-o, --output <FILE>: Output OPML file path (default:feeds.opml)
Example
Create a file urls.txt with URLs:
https://github.blog
https://stackoverflow.blog
https://www.rust-lang.org/
Run the command:
Or use the compiled binary:
Input File Format
- One URL per line
- Lines starting with
#are treated as comments and ignored - Empty lines are ignored
Example:
# Tech blogs
https://github.blog
https://stackoverflow.blog
# Programming languages
https://www.rust-lang.org/
https://go.dev/
How It Works
- Reads URLs: Parses the input file to extract URLs
- Parallel Processing: Uses Rayon to process multiple URLs simultaneously
- Feed Discovery: For each URL:
- Fetches the HTML page
- Looks for RSS/Atom feed links in the HTML
- Checks common RSS feed paths (
/feed,/rss,/feed.xml, etc.)
- Validation: Validates each discovered feed by:
- Attempting to fetch the feed
- Parsing it as RSS or Atom format
- OPML Generation: Creates a valid OPML file with all discovered and validated feeds
Dependencies
- clap: Command-line argument parsing
- rayon: Parallel processing
- reqwest: HTTP client
- scraper: HTML parsing
- opml: OPML file generation
- rss: RSS feed parsing and validation
- atom_syndication: Atom feed parsing and validation
License
This project is licensed under the MIT License - see the LICENSE file for details.