Expand description
/robots.txt for the public CMS.
The CMS has served /sitemap.xml for a long time,
and a sitemap no crawler is told about is a sitemap nobody reads: the
conventional way to advertise one is a Sitemap: line in robots.txt.
Until this module every rustango-cms site answered 404 there,
which is also why crawlers were free to spend their budget on
/cms-admin/ and the login form.
§What is served
A default that lets crawlers have the public site and keeps them out
of the admin and the auth endpoints, plus an absolute Sitemap: line
built from the request’s own scheme and host — the same
base_url the sitemap’s <loc> entries use, so the
two can never disagree about which host they are on.
§Overriding it
A tenant that needs something else — a staging site that should
disallow everything, say — stores a replacement body in the
seo site-setting scope under robots_txt, and that is served
verbatim. Nothing sensitive can leak through this: robots.txt is a
public document by definition, and the value is only ever echoed
back as text/plain.
§Wire-up
The CMS public router mounts GET /robots.txt
automatically, exactly as it does the sitemap. No registration needed.
Functions§
- handle_
robots GET /robots.txt.