Crawl-to-click Gap

Where search engines once served as the primary gateway to publishers and retailers, generative AI and large language models (LLMs) are now reshaping how traffic flows across the web. The “crawl-to-click gap” describes the growing imbalance between crawling and actual human referrals. Crawlers are automated software agents that navigate the web to index or download pages.

Traditionally, search engines used crawlers to collect content and drive users to publishers, generating traffic and conversions. Today, much of the crawling is performed to train AI models. While websites see their content being consumed at scale, fewer visitors click through to engage with the original site.

This has direct implications for eCommerce businesses. Product descriptions, blog posts, and brand content may now be used by AI systems to answer queries without sending potential customers to the retailer’s store. In effect, content that once drove traffic and sales could now benefit other platforms instead. Cloudflare offers tools to help website owners manage AI crawler access, giving brands control over which content is available for training while protecting key assets.

The crawl-to-click gap also signals an opportunity. By managing AI bot access, companies can protect valuable content and potentially explore new ways to monetise it. More interactive, personalised, or dynamic content may become increasingly important, as this type of content is less easily scraped and more likely to engage real customers.