Constant robotxt::BYTE_LIMIT

source · 
pub const BYTE_LIMIT: usize = 512_000;
Expand description

Google currently enforces a robots.txt file size limit of 500 kibibytes (KiB). See How Google interprets Robots.txt.