Crate followability 

Decide whether a link on a served page is actually followed.

Three separate signals answer that question and they have to be read together:

1. the link’s own rel attribute — nofollow, ugc and sponsored all withhold the follow;
2. the page’s <meta name="robots"> content, whose nofollow applies to every link on the page;
3. the X-Robots-Tag response header, which says the same things from outside the HTML and may be scoped to a named user agent.

Reading only the first is the common mistake: a link with no rel at all is still not followed if the response carried X-Robots-Tag: nofollow. This crate parses all three and combines them most-restrictive-wins, then tells you which signal withheld the follow.

Example

use followability::{PageDirectives, Reason, audit_link};

// rel is absent, but the header withholds the follow anyway
let page = PageDirectives::new()
    .with_x_robots_tag("nofollow");
let verdict = audit_link(None, &page, None);
assert!(!verdict.followed());
assert_eq!(verdict.reason, Some(Reason::XRobotsTagNofollow));

// rel="noopener noreferrer" is still a followed link
let clean = PageDirectives::new();
assert!(audit_link(Some("noopener noreferrer"), &clean, None).followed());

// ugc and sponsored withhold it just as nofollow does
assert_eq!(
    audit_link(Some("ugc"), &clean, None).reason,
    Some(Reason::RelUgc)
);

Parsing rules follow the HTML rel token grammar (ASCII case-insensitive, whitespace-separated; commas are also tolerated because real pages use them) and the documented robots directives, including none as shorthand for noindex, nofollow and the optional user-agent prefix on X-Robots-Tag.

No dependencies, no I/O, no unsafe. Fetching the page is your job; this crate reads what came back.

Written for the link pipeline at https://handsofflinks.com/, which re-fetches every published page and reads the attribute off the served HTML before a row may say LIVE.

Structs

LinkAudit

The combined verdict for one link on one page.

PageDirectives

Everything the page itself says, from its meta tag and its response headers.

Rel

A parsed rel attribute.

RobotsDirectives

Directives parsed from a robots meta content string or an X-Robots-Tag value.

XRobotsTag

One X-Robots-Tag header value, with its optional user-agent scope.

Enums

Reason

Why a link was not followed.

Functions

audit_link

Audit one link: its rel, the page’s directives, and the crawler you care about.