Four AI labs back one robots.txt opt-out for AI training

What happened

On October 30, 2026, four of the largest AI labs lined up behind a single way for websites to say no to AI training. OpenAI, Google, Microsoft, and Anthropic each said they will honor a shared robots.txt policy token, __usage_ai-training, so a publisher no longer has to write a separate rule for every company's crawler. Until now, opting out meant naming each crawler one by one, such as Google-Extended for Google, and repeating the work whenever a new crawler appeared.

The mechanism itself is deliberately boring, and that is the point. It uses ordinary robots.txt syntax, introduces no new directives and no new grammar, and puts the token where a crawler name normally goes. The draft's own example is:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That group expresses a preference against AI training for all paths except those under /press-kit/. A wildcard group, User-Agent: *, does not match the token, and a robots.txt file with no token group expresses no preference at all. The draft, titled Machine-Readable Training Preference Signal, was written by Kevin Kelley, a technical lead for standards at Anthropic, in the IETF's AI Preferences working group. It is dated October 30, 2026 and expires May 3, 2027. As an individual Internet-Draft, it is not endorsed by the IETF and has no formal standing in the IETF standards process yet.

The four commitments are not the same commitment on the same day. Anthropic has already started implementing support, according to Kelley. OpenAI says it aims to implement the signal in the coming months and will share implementation details later, while its existing crawler controls continue to apply. Google says it aims to support the control along with other AI providers in 2027. Microsoft gives the firmest date: it will begin honoring the new preferences on February 1, 2027, as it crawls pages.

One token, four different clocks

Google's post explains the most important interaction for its own system. If a site uses the shared token to opt out of training and does not explicitly specify Google-Extended, Google plans to interpret that as if the publisher had disallowed Google-Extended, the training control Google has offered since 2023. Company-specific controls keep working alongside the shared signal, so a site can opt out broadly and still tailor rules per provider. Google also says sites that specifically allow Google-Extended remain eligible to appear in responses and receive clicks via grounding in Gemini Apps.

Microsoft is the outlier in scope, not just timing. Beyond the shared training preference, it is adding a separate AI grounding preference: whether content may help generate AI answers in Bing, Copilot, and Microsoft 365, as distinct from training a model. It is also extending the training preference to individual pages through a meta tag or response header, and adding a Limited Grounding option that allows grounding in Microsoft's own products but not in the grounding services it sells to others. Training and grounding both default to Allow when no preference is published. Microsoft's timeline, as reported from its post, runs from temporary grounding controls in Bing Webmaster Tools on December 15, 2026, to honoring the new preferences on February 1, 2027, with sites using the older NoArchive or NoCache tags asked to switch by March 1, 2027. Temporary grounding preferences expire September 30, 2027.

What it means for you

If you run a website, the practical advice for today is unchanged: keep the company-specific rules you already have, because they are what works now. The shared token costs nothing to add and may cover you as support arrives, but adding it today does not switch anything on at Google, OpenAI, or Microsoft. Think of it as a reservation for a system still being built, not a block that is already enforced.

If you only read the web, the change is even more indirect. This signal is about training future models, not about what appears in today's search results or AI answers. Opting a site out of training does not take it out of search, and for Microsoft it explicitly does not affect eligibility for regular Bing results. The fight over AI answers, where a model quotes or summarizes a page in real time, is a separate question, and only Microsoft's grounding control addresses it directly in this announcement.

The reason publishers care anyway is leverage. For years, each lab set its own crawler names and its own rules, which put the maintenance burden on the smallest sites. A shared token flips that burden: one entry, read the same way by every participating provider. If it sticks, the next new AI crawler will not force every publisher to learn another name. If it does not stick, publishers will be back to maintaining a list, and the early adopters will have spent very little to find out.

What is still unclear: whether the draft becomes a durable standard before it expires on May 3, 2027, how each lab's implementation will handle edge cases such as page-level overrides and private agreements, and whether other providers join. Meta, for example, welcomed a common signal in reporting of the announcement but did not commit to honoring it. The first real test will not be the announcement. It will be the first publisher who adds the token, waits for a recrawl, and checks whether every lab actually respected it.

Primary sources

• Supporting a shared AI training control for websites (Google Search Central Blog) ↗
• Making it easier for websites to express AI training preferences (OpenAI) ↗
• Machine-Readable Training Preference Signal (IETF Internet-Draft) ↗
• Microsoft Bing gives publishers more control over how AI uses their content (Search Engine Land) ↗