AEDAnswer Engine Daily
← All stories

Google, Microsoft and OpenAI Plan to Honor One Shared AI Training Opt-Out in Robots.txt, but at Google the Same Line Also Takes You Out of Gemini Grounding

Sandeep Amar · November 2, 2026 · 6 min read
Google, Microsoft and OpenAI Plan to Honor One Shared AI Training Opt-Out in Robots.txt, but at Google the Same Line Also Takes You Out of Gemini Grounding
AEO · Answer Engine Daily

Google, Microsoft and OpenAI plan to honor one robots.txt line telling every participating AI company whether a site’s content can be used for training, Search Engine Journal reported on October 31. Today a site writes separate rules for each company’s crawler. None of the three reads the line yet, and an opt-out will not mean the same thing at each. At Google, it would also take a site out of Gemini Apps grounding unless the site specifically allows Google-Extended, according to Google.

Microsoft will start honoring it on February 1, 2027. Google’s crawler documentation says early 2027, and OpenAI says “coming months,” The Verge reported. Kelley’s draft expires on May 3, 2027, six months from now, unless replaced.

What does the shared line actually say?

Robots.txt is the file at a site’s root that tells crawlers what they may access. The line comes from an Internet Draft that Kevin Kelley, a technical lead for standards at Anthropic, submitted to the AI Preferences working group of the Internet Engineering Task Force (IETF), the body that sets internet standards. The IETF tracker lists it as an individual draft, which SEJ notes “does not represent official IETF standardization activity.”

The draft adds no new directive. It uses a “policy token,” __usage_ai-training, which sits where a crawler’s name goes but names a use of content, not a bot. Its own example opts a site out of training except the press kit:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

AI training, per the group’s vocabulary draft as SEJ summarises it, means using content to modify a generative AI model’s learned parameters. A chatbot learning from your articles counts; a model that only ranks or classifies content does not.

A wildcard group (User-Agent: *) never matches the token, and a file without the group states no preference, which Kelley says is true of every robots.txt file today. The token does not control crawling. This means a site must add the group deliberately; existing rules carry no training preference.

Why does the same opt-out do different things at Google and Microsoft?

Google’s John Mueller wrote that site owners “will be able to use a shared ‘policy token’ within robots.txt to set broad preferences across participating AI providers.” When a site opts out through the shared line without specifying Google-Extended, Google plans to read it “as if the publisher had disallowed Google-Extended,” Mueller’s post says.

Google-Extended covers training future Gemini models and grounding in Gemini Apps and Vertex AI, according to Google’s crawler documentation. Grounding, in Google’s description, means supplying search-index content to the model when it answers. Your article shaping a future Gemini is training; Gemini Apps pulling it for today’s question is grounding. Mueller’s post adds that sites that specifically allow Google-Extended are eligible for Gemini Apps grounding. SEJ found no documented way to refuse training but keep grounding.

This means a publisher who refuses training with the shared line alone also gives up Gemini Apps grounding. Google Search is unaffected; AI Overviews and AI Mode have a separate Search Console opt-out, the Search-versus-Gemini split I covered in Google’s AI optimization guide.

Microsoft splits the two. Its post, by Bing principal product manager Krishna Madhavan, defines “AI Training” as “the shared preference expressed in robots.txt for a site or path,” which a page can override, and adds “a separate AI Grounding preference.” It calls the working group’s “current work” “significant progress that merits adoption,” describing it as the group’s “year plus effort,” though the IETF tracker lists Kelley’s text as an individual draft.

Grounding is set per page and defaults to Allow. Disallow keeps a page out. DisallowExtended permits grounding in Copilot and Bing’s AI Mode but blocks Web IQ, Microsoft’s grounding service for other businesses. Either restriction limits brand visibility in AI answers, not Bing listings, Microsoft said. This means a publisher can refuse training and stay in Copilot answers, which Google documents no way to do for Gemini.

Microsoft is also retiring NoArchive and NoCache for these uses:

  • December 15: a temporary Bing Webmaster Tools preference opens for sites still using those tags that day and wanting out of grounding.
  • February 1, 2027: the new preferences are honored as Bing crawls.
  • March 1, 2027: the old tags stop counting. Pages without the temporary setting default to allowing grounding until recrawled, Microsoft’s FAQ says.

That leaves one month, February 1 to March 1, when Bing reads both the old tags and the new preferences, unless a site already uses a new preference, in which case Microsoft says it disregards NoArchive and NoCache on that site.

Where do OpenAI and Anthropic stand?

OpenAI said it will honor the signal for “foundational generative AI training,” and its current robots.txt policy stays. Neither its post nor its crawler documentation, checked by SEJ on October 30, says what happens when the shared line and a GPTBot rule conflict.

The Verge, crediting Search Engine Roundtable with the first report, said Anthropic “has already adopted the signal.” Kelley’s mailing-list post, as SEJ describes it, says only that Anthropic has started development and that support across all its services will take time. On October 30, SEJ found no Anthropic newsroom post and three crawler controls: ClaudeBot, Claude-User and Claude-SearchBot.

My read

Training and citation are two pipes, and each deserves its own decision. For brands, a blanket training opt-out is a poor trade: you shrink your presence in the model, and at Google, if you use the shared line without allowing Google-Extended, you lose Gemini Apps grounding too. Publishers face a business call, because their content is the product. Microsoft lets them make it cleanly. Google, for now, does not.

What is still not known?

  • Google: will it document a way to opt out of Gemini training but keep Gemini Apps grounding?
  • OpenAI: which rule wins when the shared line and GPTBot disagree?
  • Anthropic: when will support reach all its services?
  • The IETF working group: Kelley plans to propose his design as a change to the Content-Usage draft that Google’s Gary Illyes co-wrote and the group set aside. Will that happen before May 3, 2027?

Sources

More in AEO