
Tech & AI
By TrendScope Daily Team · November 2, 2026
For years, a website that wanted to keep its text and images out of AI training data had to write a separate rule for every crawler: one line for GPTBot, another for Google-Extended, another for Anthropic’s ClaudeBot, and so on, with no guarantee any of them would be read the same way twice. On Friday, October 30, 2026, that patchwork started to give way to something closer to a shared standard, as Google, OpenAI, Anthropic and Microsoft all said they would honor a single opt-out signal inside robots.txt, the plain-text file sites already use to tell crawlers what they may and may not fetch.
A single token, instead of four different rulebooks
The proposal itself is small and, on paper, unglamorous: a new policy token, written as __usage_ai-training, that a site can drop into a robots.txt group alongside the paths it does not want used to train generative models. Anthropic’s Kevin Kelley published the draft as an individual submission to the IETF, the body that oversees core internet standards, and announced it on the IETF’s AI preferences mailing list the same day. Within hours, Google, OpenAI and Microsoft had each confirmed they intended to support it, turning what could have been one company’s pitch into the closest thing the industry has had to a common data-use signal since robots.txt itself was informally standardized in the 1990s.
Crucially, the token is a statement of preference, not an access control. A site that says nothing is read as having expressed no preference either way, and private licensing agreements between a publisher and an AI company can still override the default. The IETF has not formally adopted the draft, so for now it carries the weight of an industry gentleman’s agreement rather than a binding rule — but with four of the largest AI labs and platforms publicly committed, it is already more consequential than most individual submissions ever become.
Four companies, four different timelines
What makes the announcement more interesting than a typical joint statement is that none of the four companies are implementing it on the same schedule. Anthropic, as the draft’s author, says it has already begun the engineering work and that the new token will sit alongside the crawler controls it already offers, though Kelley was candid that extending it across all of Anthropic’s services “will take us some time.” OpenAI has committed to supporting the signal “in the coming months,” while keeping its existing GPTBot controls active in the meantime.
Google’s timeline is the longest of the four: it has pencilled in support for 2027. In the interim, Google says that if a site adopts the new token without separately addressing its own Google-Extended crawler tag, it will simply treat that as an instruction to block Google-Extended too, so publishers are not left exposed in the gap. Microsoft has set a firmer date, saying it will begin honoring the preference from February 1, 2027, at which point the new token will replace the NoArchive and NoCache tags Bing has used since 2023. Microsoft is also layering in a separate, page-level setting for “grounding,” which controls whether its AI answers can draw on a page’s live content at all, a distinct question from whether that page can be used in training.
“Publishers should not have to choose between being discoverable in Bing and controlling how generative models use their work.”
— Krishna Madhavan, Principal Product Manager, Microsoft
What it does not change
Both Microsoft and Google were careful to draw a line around what the new signal actually accomplishes. Microsoft said explicitly that honoring the preference creates no licensing or payment arrangement between a publisher and the company — it is a technical courtesy, not a revenue mechanism. Google, for its part, framed the move partly in regulatory terms, noting that the proposal is compatible with the European Union’s Code of Practice for general-purpose AI models under the EU AI Act, which has pushed major labs toward clearer disclosure of what they scrape and train on.
Not every major platform has signed on in the same way. Meta’s communications chief, Andy Stone, voiced support for the idea of a common signal in a post on X, but stopped short of committing Meta to actually honoring it, leaving one of the largest AI training operators in the industry as a notable question mark. Google’s search advocate John Mueller also flagged a wrinkle for publishers chasing visibility in AI products: sites that keep Google-Extended enabled can still appear in answers generated by the Gemini app, which means the training opt-out and AI-answer visibility are not quite the same lever, even if they are easy to conflate.
Why publishers have been asking for exactly this
The announcement lands after more than two years of friction between publishers and AI labs over scraping. News organizations, image libraries and independent blogs have spent that time stacking up crawler-specific rules, legal threats and, in some cases, lawsuits, arguing that opting out of one company’s bot said nothing about another’s. A single, shared token does not resolve the underlying argument about whether training on public web content requires consent or compensation in the first place — but it does remove one of the more tedious practical objections, that there was no reliable way to say “no” to everyone at once.
For site owners who want to get ahead of the rollout rather than wait for February 2027, the practical step is straightforward once the major crawlers finish implementing support: add the __usage_ai-training group to an existing robots.txt file, decide which paths to exclude, and keep an eye on whichever crawler-specific tags — Google-Extended, GPTBot, ClaudeBot — are already in place, since none of the four companies are retiring those immediately.
Quick questions
What is the new AI training opt-out signal?
A robots.txt policy token, written as __usage_ai-training, that tells crawlers a site’s content should not be used to train generative AI.
Which companies support the AI training opt-out?
Google, OpenAI, Anthropic and Microsoft said on October 30, 2026 that they would honour it.
Is the AI opt-out an official internet standard?
Not yet. It was submitted as a draft to the IETF, the body that oversees core internet standards.
Related on TrendScope Daily
- OpenAI, Google, and Anthropic Are Quietly Building Their Own AI Regulator
- Claude Haiku 5.5 Just Got 90% Cheaper — and It’s Already Beating OpenAI’s Budget Model
- Google’s Gemini 4 Argon Lands With a Catch: Almost No One Can Use It Yet
- More Tech & AI stories: browse the Tech & AI section
Bottom line
A robots.txt token will not settle the bigger fight over whether AI labs owe publishers money for training data, and with Google not committing until 2027 and Meta still uncommitted, the “single standard” is still more aspiration than reality today. But for the first time, the four companies that run most of the web’s AI crawlers have agreed on the shape of an opt-out, which is more alignment than this fight has produced in years.
Readers who want a deeper, non-technical walkthrough of how these AI systems are actually trained — and what “opting out” can and cannot protect against — may find Ethan Mollick’s Co-Intelligence: Living and Working with AI a useful primer on the forces behind this week’s announcement.
As an Amazon Associate, TrendScope Daily earns from qualifying purchases made through links in this article.