The Google-Extended, a standalone product token designed to instruct Google via robots.txt not to utilize your website’s content for projects like Bard and Vertex AI, proves ineffective in preventing the utilization of your content for AI-answers and snapshots within the Search Generative Experience.
Google has explicitly communicated that Google-Extended does not apply to the Search Generative Experience (SGE). According to a statement from Google, “SGE is considered a Search experiment, so website administrators should continue using the Googlebot user agent in conjunction with robots.txt and the NOINDEX meta tag to control the presence of their content in search results, including experimental components such as the Search Generative Experience.”
Why doesn’t Google Extended apply to SGE? Google clarifies that SGE is an integral part of the Google Search experience, functioning as a standard search feature. Consequently, it is expected to adhere to the same rules as regular search directives. Google expounds, “The context is that AI is seamlessly integrated into Search, not tacked on, and is fundamental to how Search operates. Therefore, robots.txt is the control mechanism, allowing web publishers to regulate access to how their websites are crawled.”
Regarding AI-answers within SGE, there are instances where SGE showcases AI-generated responses, complete with website cards, even from sites that have expressly directed Google not to employ their content for AI purposes.
As an illustration, Glenn Gabe shared a screenshot of VentureBeat.com. This website has explicitly prohibited Google-Extended in its robots.txt file, yet it is still featured in an AI-generated answer within SGE.
Why is this significant? You would be mistaken if you believed that employing Google-Extended would safeguard your content from being utilized for AI-powered answers in SGE. Google-Extended is designed for applications like Bard and Vertex AI and other non-search-specific products.
If your objective is to prevent your answers from appearing in SGE, you must impose a complete block on Googlebot, a measure most website administrators are hesitant to implement.
If you’re still grappling with the intricacies and confusion, explore our monthly SEO packages and entrust the task to our skilled experts.
