Cloudflare replaces its blanket AI bot block with granular controls for search, training, and agent crawlers

Cloudflare has introduced granular controls that allow website owners to separately manage traffic from search engines, AI training models, and autonomous agents. This update replaces the previous all-or-nothing blocking method, giving administrators more precise authority over how their content is accessed by third-party scrapers. Additionally, starting September 15, 2026, the company will automatically block training and agent bots by default on pages that utilize ad-supported revenue models.