Google said on October 30, 2026 that it plans to recognise a shared robots.txt preference for AI model training in 2027. The proposed token, __usage_ai-training, is intended to let a publisher express a preference that multiple participating AI providers can read. Google’s October 30 announcement gives 2027 as a target, not a firm launch date. The signal is not yet a final IETF standard or a confirmed production control for Google.
The announcement is relevant to publishers that currently maintain separate AI crawler rules. It also creates an important distinction: a preference about using content to train a generative AI model is not the same as preventing a crawler from fetching a URL, or excluding a page from Google’s AI search features.
What was announced on October 30
In an official Search Central post, Google said it intends to support a common training preference based on work associated with the Internet Engineering Task Force’s AI Preferences (AIPREF) working group. The mechanism uses a policy token in an ordinary robots.txt User-agent group rather than introducing a new directive.
The October 30 Internet-Draft by Kevin Kelley describes how this could work for individual URL paths. The IETF Datatracker explicitly labels it an individual Internet-Draft that is not endorsed by the IETF and has no formal standing as an approved standard. Related AIPREF vocabulary and attachment specifications are also still in development.
Google’s interpretation is particularly significant. If a publisher expresses a training opt-out with the shared token and does not explicitly set Google-Extended, Google plans to treat the shared preference as though Google-Extended were disallowed. If the publisher sets a separate Google-Extended rule, that company-specific choice remains relevant. This is a stated future implementation rule, not evidence that the new token already operates in Google’s production systems.
Four providers have announced different adoption positions
The original question is no longer whether Google is the only company to speak publicly about the shared signal. On October 30, OpenAI and Microsoft also announced support plans, while Anthropic said it had begun recognising the preference. Their schedules and treatment of other AI uses are not identical.
| Provider | Published position | Boundary for publishers |
|---|---|---|
| Targets support in 2027 without a confirmed activation date. | Shared opt-out is to map to Google-Extended when no explicit Google-Extended rule exists. | |
| Microsoft Bing | Announced new AI preferences effective from February 1, 2027 as content is recrawled. | Separates AI training from AI grounding and offers additional page-level preferences. |
| OpenAI | Said it intends to implement the common signal in the coming months. | Existing crawler-specific controls remain in place. |
| Anthropic | Said on October 30 that it had begun recognising the signal, with broader rollout continuing. | Explicit ClaudeBot rules take precedence over the general preference. |
These positions come from Microsoft’s Bing announcement, OpenAI’s October 30 statement and Anthropic’s crawler guidance. A common token does not make the companies’ training, grounding and crawling policies identical. Microsoft also describes a March 1, 2027 transition deadline for publishers relying on its older NoArchive and NoCache controls. That is a Microsoft-specific operational deadline, not a common AIPREF launch date.
How the draft token differs from a crawler block
The draft uses the literal User-agent: __usage_ai-training token with familiar Allow and Disallow rules. Its illustrative whole-site training opt-out looks like this:
User-agent: __usage_ai-training
Disallow: /
This is draft syntax for a content-use preference, not a deployment instruction or a ban on all crawler requests. A narrower Disallow path or a specific Allow exception would express a different preference. The draft says crawler-specific robots.txt groups continue to control access independently.
The draft also says a wildcard User-agent: * group is not interpreted as a training preference for the shared token. If the token is absent, no preference is expressed by this mechanism. A provider that has not implemented the signal may ignore it, and a shared preference cannot revoke copies of content already collected.
Publishers should keep separate records of their crawl-access rules and their intended content-use preferences. MOCOBIN’s robots.txt best-practices guide explains the crawl-access side; its AI crawler blocking analysis examines access restrictions and allowlists. Neither mechanism should be described as a legal access barrier or a guarantee of compliance by all automated agents.
Training, grounding and Google Search remain separate decisions
Google-Extended has a wider practical consequence than the phrase ‘training opt-out’ alone suggests. Google’s crawler documentation says this product token affects both use for future Gemini model training and specified grounding in Gemini Apps and Vertex AI. Google also states that Google-Extended does not affect inclusion in ordinary Google Search and is not a Search ranking signal.
That means a publisher considering the shared training opt-out should examine its intended treatment of Gemini Apps grounding as well. Google says sites that explicitly allow Google-Extended remain eligible to be used in Gemini Apps grounding and to receive clicks, but eligibility does not promise inclusion or traffic.
For Google’s AI Overviews and AI Mode within Search, site owners must assess the relevant Search-specific guidance, including Googlebot crawl access, indexing, and snippet controls such as nosnippet and max-snippet. A training preference in robots.txt should not be presented as an interchangeable setting for Search AI visibility.
Microsoft’s new model underlines why provider-by-provider review still matters. Its announcement separates AI training preferences from AI grounding preferences. Applying a single interpretation of ‘AI opt-out’ across every platform would conceal those product differences.
What a publisher should check before changing robots.txt
A site owner does not need to rewrite a live robots.txt file solely because the October 30 announcements were published. The practical first step is to document current provider-specific rules, the content uses the organisation wants to permit, and the products where continued visibility matters.
- Inventory existing controls. Record Google-Extended, ClaudeBot and other explicit crawler groups, plus Search indexing and preview controls. Confirm what each controls before combining rules.
- Separate the desired outcome. Decide whether the concern is crawl access, generative-model training, grounding in AI answers, or search-result presentation. The new token addresses only one of those concerns directly.
- Check implementation status per provider. Anthropic has reported that it began recognising the signal; Google and OpenAI have described future support, while Microsoft has published a February 2027 effective date. Do not infer one shared launch date.
- Review documented precedence before deployment. A provider-specific rule can override or qualify a shared preference. Check the current draft, each provider’s operational documentation and how the intended paths would be interpreted.
- Keep an auditable change record. Test the resulting file and record the date, affected hosts, rules and rollback path. An unverified sample copied from a draft should not become a site-wide change by default.
Google also described the proposal as consistent with the EU Code of Practice for General Purpose AI Model Providers. That statement does not, by itself, establish legal compliance or make an unfinished IETF draft binding on every AI company. The next milestones are provider implementation documentation, the state of the IETF drafts and any changes to product-specific controls.
- Google Search Central, October 30, 2026: Shared AI training control announcement
- Google Crawling Infrastructure: Google-Extended scope and its Search boundaries
- IETF Datatracker: Training preference signal, individual Internet-Draft
- IETF AIPREF working group: Active documents
- Microsoft Bing, October 30, 2026: Training and grounding preferences
- OpenAI, October 30, 2026: Shared training preferences
- Anthropic: Website training preferences and ClaudeBot precedence
- Google Search Central: AI features in Search and site controls








