rss-miner 0.1.0

CLI tool that finds RSS/Atom feeds from URLs and generates OPML.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96

# rss-miner

A CLI tool that finds RSS feeds from URLs and generates a valid OPML file.

## Features

- 🔍 **Parallel Processing**: Uses Rayon to process multiple URLs concurrently
- ✅ **RSS Feed Validation**: Validates RSS/Atom feeds before including them
- 📝 **OPML Generation**: Creates a valid OPML file compatible with feed readers
- 🎯 **Auto-Discovery**: Finds RSS feeds in HTML link tags and common feed paths
- 🛡️ **Error Handling**: Robust error handling with detailed feedback

## Installation

```bash
cargo build --release
```

## Usage

```bash
rss-miner --input <INPUT_FILE> [--output <OUTPUT_FILE>]
```

### Arguments

- `-i, --input <FILE>`: Input file containing URLs (one per line, required)
- `-o, --output <FILE>`: Output OPML file path (default: `feeds.opml`)

### Example

Create a file `urls.txt` with URLs:

```
https://github.blog
https://stackoverflow.blog
https://www.rust-lang.org/
```

Run the command:

```bash
cargo run -- --input urls.txt --output feeds.opml
```

Or use the compiled binary:

```bash
./target/release/rss-miner --input urls.txt --output feeds.opml
```

### Input File Format

- One URL per line
- Lines starting with `#` are treated as comments and ignored
- Empty lines are ignored

Example:

```
# Tech blogs
https://github.blog
https://stackoverflow.blog

# Programming languages
https://www.rust-lang.org/
https://go.dev/
```

## How It Works

1. **Reads URLs**: Parses the input file to extract URLs
2. **Parallel Processing**: Uses Rayon to process multiple URLs simultaneously
3. **Feed Discovery**: For each URL:
   - Fetches the HTML page
   - Looks for RSS/Atom feed links in the HTML
   - Checks common RSS feed paths (`/feed`, `/rss`, `/feed.xml`, etc.)
4. **Validation**: Validates each discovered feed by:
   - Attempting to fetch the feed
   - Parsing it as RSS or Atom format
5. **OPML Generation**: Creates a valid OPML file with all discovered and validated feeds

## Dependencies

- **clap**: Command-line argument parsing
- **rayon**: Parallel processing
- **reqwest**: HTTP client
- **scraper**: HTML parsing
- **opml**: OPML file generation
- **rss**: RSS feed parsing and validation
- **atom_syndication**: Atom feed parsing and validation

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.