Google, Microsoft, OpenAI and Anthropic are moving towards a shared standard that would let publishers state their AI-training preferences through a single machine-readable robots.txt token. The initiative builds on the IETF's developing AIPREF specification. Instead of maintaining a separate directive for each AI company's crawler, a site could express one policy that applies across competing AI ecosystems. Google says it plans to support the signal in early 2027. According to the reporting, the change is primarily about model training, not necessarily whether search engines may crawl or cite content in AI answers. Source: Google Search Central announcement.
Why it matters: this could become the first broadly shared control for publishers managing how their content is used by AI systems. Fragmented, bot-by-bot opt-out rules have been a real headache, and a common signal would cut that down considerably. The caveat is that the standard is voluntary and still a work in progress, so its practical effect will depend on each provider actually adopting and honouring it. Search Engine Journal has been tracking the wider debate around AI crawler controls.
What this means for getting cited by AI: training preferences and citation eligibility are not the same thing. Opting out of training does not automatically remove you from AI-generated answers, and the reporting so far doesn't establish that a training opt-out would affect citation at all. Our advice is to treat these signals as separate levers. Audit which AI crawlers can reach your content, decide your training policy deliberately, and check whether your pages are actually being surfaced in AI answers. If you want to know where you stand, our GEO audit at Citeify maps crawler access against citation visibility so you can make that call with data rather than guesswork.