Google, Microsoft and OpenAI Back a Shared robots.txt Signal to Opt Out of Super Intelligence (SI) Training
Google announced on October 30, in a post on its official Search Central blog, that it plans to support a shared super intelligence (SI) training control in robots.txt: websites will be able to use a single shared policy token to tell multiple participating SI providers at once whether their content may be used for model training, instead of setting separate rules for each company. According to the post, written by John Mueller, the mechanism builds on work by the AIPREF working group at the Internet Engineering Task Force (IETF).
The mechanism is set out in an Internet-Draft titled Machine-Readable Training Preference Signal, submitted the same day and authored by Kevin Kelley of Anthropic; its core is a policy token named __usage_ai-training. Google says that where a site opts out of training through the shared token and does not explicitly specify Google-Extended, Google plans to interpret that as disallowing Google-Extended, the control it has offered since 2023, which covers training of future Gemini models and grounding in Gemini Apps and on Vertex AI. Google aims to support the new control in 2027.
OpenAI made a parallel commitment the same day, saying it will honor the shared signal, aims to implement it in the coming months, and will keep its existing crawler controls in place. Microsoft's Bing said it will honor the shared training preference from February 1, 2027, and is adding a separate page-level grounding preference; from March 1, 2027, it will stop reading NoArchive and NoCache as training or grounding signals, so sites still relying on those tags will need to switch to the new settings.
The signal covers training only and does not affect crawling or inclusion in search. It also remains an individually submitted Internet-Draft rather than a formal IETF standard for now, and existing company-specific controls continue to work during the transition.
Source: GSM · USA (original Chinese edition)
Photo: 图：谷歌总部 Googleplex（加利福尼亚州 Mountain View）。Grendelkhan 摄，CC BY-SA 3.0，经 Wikimedia Commons。 / The Googleplex, Google's headquarters in Mountain View, California. Photo by Grendelkhan, CC BY-SA 3.0, via Wikimedia Commons.
