followability 0.1.0

Decide whether a link is followed: parse rel token lists, robots meta contents and X-Robots-Tag headers, and combine them into one verdict.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

# followability

Decide whether a link on a served page is actually followed, in dependency-free Rust.

Three signals answer that question and they have to be read together:

1. the link's own `rel` attribute — `nofollow`, `ugc` and `sponsored` all withhold the follow;
2. the page's `<meta name="robots">` content, whose `nofollow` applies to every link on the page;
3. the `X-Robots-Tag` response header, which says the same things from outside the HTML and
   may be scoped to a named user agent.

Reading only the first is the common mistake: a link with no `rel` at all is still not followed
if the response carried `X-Robots-Tag: nofollow`.

```rust
use followability::{audit_link, PageDirectives, Reason};

// rel="noopener noreferrer" is a followed link
assert!(audit_link(Some("noopener noreferrer"), &PageDirectives::new(), None).followed());

// a bare anchor is still not followed if the header says so
let page = PageDirectives::new().with_x_robots_tag("nofollow");
let v = audit_link(None, &page, None);
assert!(!v.followed());
assert_eq!(v.reason, Some(Reason::XRobotsTagNofollow));

// noindex does not withhold the follow — it is a separate question
let page = PageDirectives::new().with_meta_robots("noindex, follow");
let v = audit_link(None, &page, None);
assert!(v.followed());
assert!(!v.page_indexable);

// scoped headers only bind their agent
let page = PageDirectives::new().with_x_robots_tag("bingbot: nofollow");
assert!(audit_link(None, &page, Some("googlebot")).followed());
assert!(!audit_link(None, &page, Some("bingbot")).followed());
```

Handled: the HTML `rel` token grammar (ASCII case-insensitive, whitespace-separated, commas
tolerated), `none` as shorthand for `noindex, nofollow`, contradictions resolved
most-restrictive-wins, valued directives such as `max-snippet:-1` whose colon is not a
user-agent separator, and multiple `X-Robots-Tag` headers merged per agent.

## Install

```toml
[dependencies]
followability = "0.1"
```

`#![forbid(unsafe_code)]`, no dependencies, MSRV 1.63. Fetching the page is your job; this
crate reads what came back.

## Licence

MIT OR Apache-2.0.

Written for the link pipeline at <https://handsofflinks.com/>.