Google Alters Robots.txt Policy: Ignoring Unrecognized Fields
Google Limits Robots.txt Support to Four Fields, Clarifies Stance on Unsupported Directives. Google now officially supports only four specific fields in robots.txt. Any unsupported directives will be ignored, so it’s a good idea to audit your robots.txt files accordingly. In its latest update to the Search...