Constant robotxt::BYTE_LIMIT
source · pub const BYTE_LIMIT: usize = 512_000;
Expand description
Google currently enforces a robots.txt
file size limit of 500 kibibytes (KiB).
See How Google interprets Robots.txt.
pub const BYTE_LIMIT: usize = 512_000;
Google currently enforces a robots.txt
file size limit of 500 kibibytes (KiB).
See How Google interprets Robots.txt.