Module ai_bots 

Registry of known AI bot user-agents and crawler identification.

Contains a static list of AI crawlers (GPTBot, ClaudeBot, etc.) with metadata for robots.txt analysis and AI accessibility scoring.

Structs

AiBot

Represents an AI crawler/bot that may access web content.

AiBotRegistry

Registry of known AI bots for robots.txt analysis.

Statics

AI_BOTS

Default list of known AI bots.

Functions

robots_txt_disallows_bot

Parse robots.txt content and check if a specific bot is disallowed.