🔍 Read the full analysis: What To Expect When You Set The Shared AI Training Opt-Out on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI, Google, Microsoft and Anthropic have announced support for a shared robots.txt signal asking them not to use a website’s content to train generative AI models. The signal is voluntary, does not block crawling or AI answers based on live web content, and will take effect on different schedules; Meta has expressed support without committing to honor it.
OpenAI, Google, Microsoft and Anthropic said on October 30 that they will honor a shared robots.txt signal asking them not to use a website’s content to train generative AI models. The move could simplify how publishers communicate that preference, but the signal is voluntary, not an enforceable block, and each company has set a different implementation timetable.
The proposed rule uses the user-agent name __usage_ai-training with Disallow: / in a site’s robots.txt file. It is described in an individual Internet-Draft titled “Machine-Readable Training Preference Signal,” published October 30 by Kevin Kelley of Anthropic. The draft is connected to the Internet Engineering Task Force’s AI Preferences working group, but it is not an adopted internet standard.
The companies’ commitments differ. Anthropic says it has started implementation, though it says support will take time to reach its services. OpenAI aims to implement the signal “in the coming months.” Google says it aims to support it in 2027; its crawler documentation specifies early 2027. Microsoft plans to honor it from February 1, 2027, as it crawls websites.
Microsoft is also introducing a separate page-level preference for AI Grounding, which concerns using web content to inform AI answers in Bing, Copilot and Microsoft 365. Microsoft says that preference and the training signal take effect February 1, 2027. Its existing NoArchive and NoCache preferences will stop counting as training or grounding preferences after March 1, 2027. Existing crawler rules such as GPTBot, Google-Extended and ClaudeBot remain available.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
A simpler signal, with limits
For website operators, a shared rule could reduce the need to maintain separate instructions for each participating company. Under Google’s stated approach, if a site uses the new signal but has no Google-Extended rule, Google plans to treat the shared opt-out as if the site had disallowed Google-Extended. That may make one preference easier to communicate, although operators should retain company-specific controls while providers roll out support.
The distinction between training and other uses matters. The shared rule is a request about training generative AI models; it does not, by itself, tell companies to stop crawling a site or prevent content from being used to generate answers from current web pages. Microsoft’s separate grounding control illustrates that those are different choices. Microsoft also says neither preference guarantees search inclusion, traffic or payment.
For publishers, the signal offers a common way to state a preference, not a technical guarantee that every company will comply. The announcements could also influence how other AI services handle publisher instructions, but support from companies beyond the four named providers has not been established.
Products Worth Considering
Sonos Era 100 SL Compact Microphone-Free Speaker – Black
Bose QuietComfort Ultra Wireless Noise-Cancelling Headphones with Spatial Audio, Black
DJI Osmo Mobile 8 Standard Combo Gimbal Stabilizer for Phone
As an affiliate, we earn on qualifying purchases.
Until now, site owners commonly used different crawler-specific rules to express preferences to different AI companies. Examples include GPTBot for OpenAI, Google-Extended for Google and ClaudeBot for Anthropic. Microsoft’s NoArchive and NoCache page tags have also been used for particular content preferences. The new draft proposes a shared training-specific token alongside those existing controls.
The draft is an individual Internet-Draft, not a formal IETF standard. At a working-group meeting on October 21, a chair said the group had not declared consensus on anything. The document’s connection to the IETF working group therefore does not mean the group or the IETF has approved it. Company announcements describe their own intended policies and implementation plans.
Meta spokesperson Andy Stone said on X that Meta “strongly support[s] publishers’ ability to express preferences” and called a common signal “an important step.” That statement signals support for the idea, but does not commit Meta to honoring the rule. The source report also said it found no statements on the signal from Apple, Amazon, xAI or Perplexity.
““has already started implementing support for this draft””
— Anthropic
Adoption and enforcement remain open
How consistently the signal will be honored remains uncertain because the draft is a request, not a mechanism that technically prevents access or use. The companies have announced intentions, but their rollout dates differ, and Anthropic’s services may not all support it at once. The cited material does not establish how compliance will be monitored or what recourse a publisher would have if content were used contrary to the preference.
It is also unclear whether Meta will implement the signal or whether other AI providers will adopt it. The draft’s status in the IETF process may change, but the available account says it had no formal standing in that process at publication. The signal does not remove material from models already trained, and its training scope should not be read as covering AI answers that draw on live web content.
Watch the rollout dates
Website operators can add the shared rule if they want to express a preference against training use, while keeping existing company-specific rules until providers have implemented the common signal. They should check each provider’s documentation for current controls rather than assume that an announced plan is already active. Google’s stated target is early 2027; Microsoft’s shared training and grounding preferences begin February 1, 2027, with its stated change to NoArchive and NoCache preferences on March 1, 2027.
The next developments to watch are whether Anthropic completes support across its services, OpenAI and Google meet their stated timelines, and Meta or other providers make explicit commitments. The draft’s progress through the IETF working group is another separate milestone: publication alone does not make the proposal an adopted standard.
Key Questions
What does the shared AI training opt-out do?
It is a robots.txt preference asking participating companies not to use a website’s content to train generative AI models. The proposed rule uses User-Agent: __usage_ai-training and Disallow: /.
Does the rule block AI companies from crawling a website?
No. The signal expresses a preference about training use; it does not itself block crawling. Existing crawler-specific controls remain available.
Does opting out stop a website from appearing in AI answers?
Not by itself. The shared signal addresses model training, not AI answers informed by live web content. Microsoft has announced a separate AI Grounding preference for some of its services.
When will the companies honor the signal?
Anthropic says implementation has started; OpenAI aims for the coming months; Google targets 2027, with documentation saying early 2027; and Microsoft’s stated start date is February 1, 2027. These are different schedules, not a single shared launch date.
Has Meta committed to honoring the opt-out?
No commitment is established in the cited material. Meta spokesperson Andy Stone expressed support for publishers’ ability to state preferences, but did not say whether Meta would honor this specific signal.
Source: ThorstenMeyerAI.com
