Skip to main content

Module robots

Module robots 

Source
Expand description

/robots.txt for the public CMS.

The CMS has served /sitemap.xml for a long time, and a sitemap no crawler is told about is a sitemap nobody reads: the conventional way to advertise one is a Sitemap: line in robots.txt. Until this module every rustango-cms site answered 404 there, which is also why crawlers were free to spend their budget on /cms-admin/ and the login form.

§What is served

A default that lets crawlers have the public site and keeps them out of the admin and the auth endpoints, plus an absolute Sitemap: line built from the request’s own scheme and host — the same base_url the sitemap’s <loc> entries use, so the two can never disagree about which host they are on.

§Overriding it

A tenant that needs something else — a staging site that should disallow everything, say — stores a replacement body in the seo site-setting scope under robots_txt, and that is served verbatim. Nothing sensitive can leak through this: robots.txt is a public document by definition, and the value is only ever echoed back as text/plain.

§Wire-up

The CMS public router mounts GET /robots.txt automatically, exactly as it does the sitemap. No registration needed.

Functions§

handle_robots
GET /robots.txt.