followability

Decide whether a link on a served page is actually followed, in dependency-free Rust.

Three signals answer that question and they have to be read together:

1. the link's own rel attribute — nofollow, ugc and sponsored all withhold the follow;
2. the page's <meta name="robots"> content, whose nofollow applies to every link on the page;
3. the X-Robots-Tag response header, which says the same things from outside the HTML and may be scoped to a named user agent.

Reading only the first is the common mistake: a link with no rel at all is still not followed if the response carried X-Robots-Tag: nofollow.

use followability::{audit_link, PageDirectives, Reason};

// rel="noopener noreferrer" is a followed link
assert!(audit_link(Some("noopener noreferrer"), &PageDirectives::new(), None).followed());

// a bare anchor is still not followed if the header says so
let page = PageDirectives::new().with_x_robots_tag("nofollow");
let v = audit_link(None, &page, None);
assert!(!v.followed());
assert_eq!(v.reason, Some(Reason::XRobotsTagNofollow));

// noindex does not withhold the follow — it is a separate question
let page = PageDirectives::new().with_meta_robots("noindex, follow");
let v = audit_link(None, &page, None);
assert!(v.followed());
assert!(!v.page_indexable);

// scoped headers only bind their agent
let page = PageDirectives::new().with_x_robots_tag("bingbot: nofollow");
assert!(audit_link(None, &page, Some("googlebot")).followed());
assert!(!audit_link(None, &page, Some("bingbot")).followed());

Handled: the HTML rel token grammar (ASCII case-insensitive, whitespace-separated, commas tolerated), none as shorthand for noindex, nofollow, contradictions resolved most-restrictive-wins, valued directives such as max-snippet:-1 whose colon is not a user-agent separator, and multiple X-Robots-Tag headers merged per agent.

Install

[dependencies]
followability = "0.1"

#![forbid(unsafe_code)], no dependencies, MSRV 1.63. Fetching the page is your job; this crate reads what came back.

Licence

MIT OR Apache-2.0.

Written for the link pipeline at https://handsofflinks.com/.