Making it easier for websites to express AI training preferences

OpenAI, Google, Microsoft, and Anthropic have committed to adopting a unified robots.txt signal that allows website operators to express their AI training preferences. This standardized approach lets publishers opt out of data scraping for multiple AI companies simultaneously rather than managing individual requests for each provider. By streamlining these controls, the initiative simplifies how site owners protect their content from being used in model training, potentially reducing the technical burden for publishers seeking to manage their digital footprint across the industry.