Cloud flare has released its sixth annual annual review, providing an in-depth look at the state of the internet, security and the booming world of AI. The report highlights major differences in web crawling activity. According to the data, Googlebot crawled significantly more websites than its AI-native competitors. Specifically, Google’s bot visited 200 times more pages than PerplexityBot.
This comprehensive Cloudflare review draws data from a vast network spanning over 330 cities in 125 countries. The results show that Googlebot reached 11.6% of unique web pages at the end of 2025. In comparison GPTBot by OpenAI achieved only 3.6%, while PerplexityBot searched only 0.06%. The report suggests that this dominance exists because Google takes a dual approach. Googlebot crawls for both search engine indexing and AI model training. As a result, web publishers cannot block Google’s AI training without also removing their websites Google search results.
Cloudflare Review: Key Takeaways from the 2025 Report
Beyond AI crawling, the report provides key data on global connectivity and security threats. Here are the specific highlights:
- Traffic growth: Global internet traffic increased 19% year-on-year, with a significant acceleration after mid-August.
- Security changes: For the first time, civil society and nonprofit organizations became the most targeted sector, accounting for 23.2% of attacks in July.
- Blocking trends: Publishers often block AI-specific bots like GPTBot via robots.txt files, while Googlebot and Bingbot usually have partial restrictions to protect login pages.
- Crawl-to-refer ratios: Anthropic showed the highest crawl to referral ratio, meaning content is heavily scanned but very few users are redirected back to the source websites.
- User action crawling: Bots that act on behalf of users (e.g. ChatGPT answering a specific question) have grown more than fifteenfold over the year.
Also read: Podcast Statistics: Market Growth, Trends and Success Strategies
The report also notes a shift in traffic composition. While human-generated traffic continues to dominate, non-AI bot traffic is close behind. Through December, humans generated 47% of requests, compared to 44% from bots. As AI tools continue to evolve, this is the case Cloudflare review serves as an important foundation for understanding how these automated agents interact with the open web.
More news to read: Publisher AI partnerships: Google strengthens relationships
AI agents are changing SEO: Marie Haynes calls for action
