robotstxt-rs 0.1.4

An intuitive Rust library for aquiring, parsing and querying robots.txt files.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120

# robotstxt-rs

[![Crates.io](https://img.shields.io/crates/v/robotstxt-rs.svg)](https://crates.io/crates/robotstxt-rs)
[![Documentation](https://docs.rs/robotstxt-rs/badge.svg)](https://docs.rs/robotstxt-rs)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

An intuitive Rust library for aquiring, parsing and querying robots.txt files.

## Installation

Add this to your `Cargo.toml`:

```toml
[dependencies]
robotstxt-rs = "0.1.4"
```

## Usage Examples

### Fetch from URL (async)

```rust
use robotstxt_rs::RobotsTxt;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Fetch robots.txt from a URL
    let robots = RobotsTxt::from_url("https://example.com/robots.txt").await?;

    // Check if a user-agent can fetch a path
    if robots.can_fetch("Googlebot", "/admin/secret") {
        println!("Allowed to crawl!");
    } else {
        println!("Blocked by robots.txt");
    }

    Ok(())
}
```

### Parse from string

```rust
use robotstxt_rs::RobotsTxt;

fn main() {
    let content = r#"
User-agent: *
Disallow: /admin/
Allow: /public/

User-agent: Googlebot
Disallow: /private/

Sitemap: https://example.com/sitemap.xml
# This is a comment
    "#;

    let robots = RobotsTxt::parse(content);

    // Check access for different user-agents
    println!("Can Mozilla access /public/? {}", robots.can_fetch("Mozilla", "/public/page.html"));
    println!("Can Mozilla access /admin/? {}", robots.can_fetch("Mozilla", "/admin/panel"));
    println!("Can Googlebot access /private/? {}", robots.can_fetch("Googlebot", "/private/data"));

    // Get sitemaps
    for sitemap in robots.get_sitemaps() {
        println!("Sitemap: {}", sitemap);
    }

    // Get comments
    for comment in robots.get_comments() {
        println!("Comment: {}", comment);
    }

    // Get all rules
    for (user_agent, rule) in robots.get_rules() {
        println!("User-agent: {}", user_agent);
        println!("  Allowed: {:?}", rule.allowed);
        println!("  Disallowed: {:?}", rule.disallowed);
    }

    // Get rules for a specific user-agent
    if let Some(rule) = robots.get_rule("Googlebot") {
        println!("Googlebot allowed paths: {:?}", rule.allowed);
        println!("Googlebot disallowed paths: {:?}", rule.disallowed);
    }

    // Get domain (if parsed from URL)
    if let Some(domain) = robots.get_domain() {
        println!("Domain: {}", domain);
    }
}
```

### Parse with custom domain

```rust
use robotstxt_rs::RobotsTxt;

fn main() {
    let content = r#"
User-agent: *
Disallow: /admin/
    "#;

    // Parse with a specific domain
    let robots = RobotsTxt::parse_with_domain(content, Some("example.com".to_string()));

    if let Some(domain) = robots.get_domain() {
        println!("Domain: {}", domain);
    }
}
```

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

Copyright (c) 2025