Refusing AI training in robots.txt now has a single line every major company recognizes. On October 30, 2026, OpenAI, Google, Microsoft, and Anthropic announced same-day support for the shared signal __usage_ai-training. Adding two lines to robots.txt states your position — but each company rolls it out on its own schedule, so the old rules still need to stay in place until then.
Get the two lines right first
User-Agent: __usage_ai-training
Disallow: /It can also apply to only part of the site, or allow specific directories — for example, add another line, Allow: /press-kit/. Note that a wildcard rule such as User-Agent: * doesn't count: the token __usage_ai-training must be written out explicitly, and a single wrong character means it doesn't apply at all. The signal comes from an Internet-Draft by the IETF AI Preferences Working Group, drafted by Anthropic engineer Kevin Kelley, and the four companies announced support on the same day so that site owners only have to write it once for it to work everywhere.
Each company's rollout timeline
| Company | Rollout |
|---|---|
| Microsoft | Crawling by the new preference from February 1, 2027 |
| Aiming to support it during 2027 | |
| OpenAI | Implementing within the coming months |
Until each company rolls it out, the old rules — GPTBot, Google-Extended, ClaudeBot, and others — still need to stay. Removing them early means giving permission during the transition.
Limits and misconceptions
This signal only covers training: it doesn't block crawling, doesn't affect search indexing, and doesn't stop AI answers from citing your pages. Microsoft has a separate page-level AI Grounding preference that distinguishes training from citation in answers. On Google's side, note this: if you use the shared signal without mentioning Google-Extended separately, it will be treated as blocking Google-Extended too — to keep appearing in Gemini answers, you need an explicit Allow. The draft is also voluntary in nature and has no formal IETF endorsement, and Meta, Apple, Amazon, and xAI have not yet committed to supporting it.
