Major AI companies are backing a unified method for website owners to block AI scrapers. According to Search Engine Roundtable and The Verge, OpenAI, Google, Microsoft, and Anthropic have agreed to support a shared web signal that allows publishers to apply their AI preferences across multiple tech companies at once, rather than blocking each crawler individually.
Until now, site creators who wanted to keep their writing, images, or articles out of AI training datasets had to write separate blocking rules for every individual AI bot. The new agreement attempts to streamline that process into a single standard.
How the shared opt-out standard works
The standard was proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force. According to The Verge, the system adds a policy token to a website’s robots.txt file, which is a small public text file that websites use to give instructions to automated web crawlers.
Under the new standard, when a participating AI company sends out its automated crawler to gather web data, encountering this policy token instructs the crawler to back off and restrict access to the site.
Rollout timelines and missing support
The shared standard is not yet active across every major lab. Anthropic has already adopted the signal. OpenAI told news outlets it will implement the standard in the coming months, while Google and Microsoft plan to roll out support next year.
Support among other big tech players remains uncertain. According to The Verge, Meta communications head Andy Stone expressed support for creating a common signal, but Meta has not confirmed whether it will actually honor the token when scraping web data.
Website owners have previously relied on other tactics to deal with AI crawlers. Security provider Cloudflare blocks AI crawlers by default, and initiatives like the Really Simple Licensing standard attempt to let publishers charge crawlers that visit their pages.
Why this standard still leaves gaps in publisher control
While a shared standard makes managing web files easier, it does not give website owners absolute control over their content.
First, rules inside a robots.txt file rely entirely on voluntary compliance. Major tech firms like OpenAI and Google may honor the token, but smaller companies, independent scrapers, or rogue bots can ignore the instruction and scrape pages anyway.
Second, companies are separating search indexing from AI training. Microsoft stated it will offer granular controls so publishers can opt out of AI model training or AI grounding—where a model pulls live web information to factualize its answers—without losing their rankings in Bing search results. Google offers similar settings, allowing sites to block AI training without hiding their pages from Google Search results. To learn more about how grounding stops models from inventing false claims, see our guide on why AI chatbots make things up.
Finally, blocking a crawler today only prevents future scraping. It does not retroactively remove a site’s text or photos from the giant datasets that AI firms spent years collecting to train their existing tools.
What this means for site owners today
For everyday bloggers, online store owners, and content creators, the shared standard will eventually reduce the headache of updating website configuration files every time a new AI company launches a crawler. The option is completely free to use since editing a robots.txt file costs nothing.
However, because full implementation across OpenAI, Google, and Microsoft will stretch into next year, publishers will still need to keep their existing individual blocking rules active in the meantime.