Cloudflare's Pay Per Crawl: What It Means for Your Site

Starting September 15, 2026, Cloudflare will block AI training and agent crawlers by default on ad-supported pages. AI companies can still get in, but only if they pay through Cloudflare's Pay Per Crawl marketplace. If you run a small business website, that block-allow-or-charge decision now lives in a dashboard setting most owners have never opened.

This isn't a future problem. It's a setting that already exists, sitting on default, deciding right now whether ChatGPT, Claude, and Gemini can read your site, and whether you get anything back for it.

What Changed on September 15, 2026?

Cloudflare's new default blocks AI training and AI agent crawlers on any page that carries ads, unless the crawler's owner pays.

The block applies to what Cloudflare calls "mixed-use crawlers," bots that combine search indexing, AI agent actions, and AI model training in one crawler. Cloudflare names Googlebot, Applebot, and Bingbot as examples (Cloudflare, "Your site, your rules"). Search crawling stays allowed by default. Training and agent crawling do not. Cloudflare says it's encouraging those companies to split their bots into three separate crawlers, so site owners can allow search while still blocking training.

The new default applies to new Cloudflare customers, new sites on existing accounts, and every free-tier account. If you already run a paid Cloudflare account with live sites, your current settings stay put until you change them.

This builds on Pay Per Crawl, the marketplace Cloudflare opened in private beta on July 1, 2025. It lets site owners charge AI crawlers a per-request fee using the HTTP 402 "Payment Required" status code (Cloudflare, "Introducing pay per crawl"). That program is now expanding into "Pay Per Use," which pays publishers when their content actually shows up in an AI answer, not just when a crawler fetches the page. Cloudflare's first two named partners on Pay Per Use are Ceramic.ai and You.com (TechCrunch, July 1, 2026).

Why Is Cloudflare Doing This Now?

Cloudflare is acting because bots now outnumber humans on the open web, and AI crawlers take far more than they give back.

Cloudflare Radar data shows bots made up 57.5% of all HTTP requests to HTML content in early June 2026, against 42.5% from humans. Cloudflare hadn't expected that crossover until 2027 (Forbes, June 4, 2026).

The imbalance is clearest in Cloudflare's own "crawl-to-refer ratio." It measures how many times a platform crawls a page for every visitor it actually sends back to that page, comparing January 2025 to July 2025:

• Anthropic (Claude): 286,930:1 in January, down to 38,065:1 by July. Still extreme.
• OpenAI (GPTBot): 1,217:1 in January, 1,091:1 by July. Roughly flat.
• Perplexity: 54:1 in January, worsening to 194:1 by July.
• Google: 3.8:1 in January, worsening to 5.4:1 by July.

Source: Cloudflare, "The crawl-to-click gap".

Cloudflare's own read on that data is blunt. Training now drives nearly 80% of AI bot activity, while referrals to publishers, especially from Google, are falling. As the company puts it: "creators face a paradox: feeding AI systems without gaining traffic in return" (Cloudflare, "The crawl-to-click gap").

Does This Actually Affect Your Small Business Site?

Probably not through the September 15 default itself, but the underlying decision applies to you either way.

The new default only changes behavior on pages that carry ads. A typical small business site without a programmatic ad network won't get auto-blocked by this specific change. But the setting Cloudflare is drawing attention to, which bots can read your site and on what terms, already applies to every website, ad-supported or not.

If your site runs on Cloudflare, you already have an AI Crawl Control panel. It lists every AI crawler hitting your domain and lets you allow, block, or charge each one on its own. Most small business owners have never opened it. If you're not on Cloudflare, the same choice sits in your robots.txt file, where GPTBot, ClaudeBot, PerplexityBot, and Google-Extended are either allowed, blocked, or simply never addressed.

Should You Block AI Bots, Allow Them Free, or Charge Through Pay Per Crawl?

There's no single right answer. Each of the three options costs you something different.

• Block everything. You stop the scraping and protect bandwidth. But you also disappear from AI answers entirely. If GPTBot or ClaudeBot can't read your site, they can't cite it when a customer asks ChatGPT or Claude who does what you do. We've written about why that access matters for getting cited in AI search results.
• Allow everything free. This is what most sites run today, and it's exactly the arrangement the crawl-to-refer numbers above describe: heavy crawling, little traffic back. You stay eligible for AI citations, but you're not compensated for what you feed those answers.
• Charge through Pay Per Crawl or Pay Per Use. Set a price per request, or opt into the model that pays when your content is actually used in an answer. Adoption is still thin (Cloudflare currently lists two named partners), so don't expect meaningful revenue from Pay Per Crawl this quarter. Treat it as a switch worth flipping on, not a new income line.

Most small businesses will land in the middle. Allow the AI crawlers that plausibly send customers your way, the ones tied to AI search visibility, and block or price the ones that only take.

What Does Gemini Spark Have to Do With Any of This?

It's proof that "AI bot traffic" no longer means only model training.

Google shipped Gemini Spark on Mac in beta on July 1, 2026. It's an AI agent that books restaurant tables, orders groceries, and completes tasks across apps like Instacart, OpenTable, and Zillow Rentals on a person's behalf (TechCrunch, July 1, 2026). Every one of those actions is a bot request landing on somebody's website or booking system, not a human clicking through a menu.

That's what flat "block all bots" advice misses. Some AI traffic hitting your site now is an agent trying to book your service or check your hours for a real customer, not a crawler training a model. Blocking indiscriminately can cut off a genuine booking attempt along with the training crawler you meant to stop. It's exactly why Cloudflare's new default separates search, agent, and training instead of treating "AI bot" as one bucket.

How to Check and Set Your AI Bot Policy This Week

Five steps, and none of them require a developer.

1. Find out if you're on Cloudflare. Check your DNS provider or ask whoever manages your hosting. If your nameservers point to Cloudflare, you already have this control.
2. Open Bot Management or AI Crawl Control in the Cloudflare dashboard. Review what's set for Search, AI Training, and AI Agent categories on each domain.
3. If you're not on Cloudflare, open your robots.txt file. Check whether GPTBot, ClaudeBot, PerplexityBot, and Google-Extended are allowed, blocked, or simply absent (absent usually means allowed).
4. Decide per category, not per bot. Keep search and citation-relevant crawling open if AI visibility matters to your business. Block or price pure training crawlers that offer no plausible referral value.
5. Revisit this quarterly. The rules, the named partners, and the pricing tools are all moving faster than a typical website review cycle.

FAQ

What is Cloudflare's Pay Per Crawl?

Pay Per Crawl is a marketplace Cloudflare launched in private beta on July 1, 2025. It lets website owners charge AI crawlers a per-request fee using the HTTP 402 "Payment Required" status code, instead of allowing free scraping or blocking access outright.

What changes on September 15, 2026?

Cloudflare's default settings will block AI training and AI agent crawlers on any page that displays ads, for new customers, new sites, and free-tier accounts. Search crawling stays allowed by default. Existing paid accounts keep their current settings unless changed.

Does this affect my small business website if I don't run ads?

The September default specifically targets ad-supported pages, so a typical small business site may not be auto-blocked. But the crawler-access decision behind it, which AI bots can read your site and on what terms, already applies to every website through robots.txt or a host's bot-management settings.

Should I block AI bots from my website?

It depends on whether AI visibility matters to your business. Blocking protects bandwidth and proprietary content but removes you from AI-generated answers on ChatGPT, Claude, Gemini, and Perplexity. Most businesses do better allowing crawlers tied to search and citation while blocking or pricing pure training crawlers.

What is a crawl-to-refer ratio?

It's the number of times an AI platform crawls a page compared with how often it sends a visitor back to that page. Cloudflare's own data shows this ratio topped 280,000 crawls per referral for one platform in early 2025, the imbalance now driving its new policy.

Sources

• Cloudflare: "Your site, your rules: new AI traffic options for all customers."
• Cloudflare: "Introducing pay per crawl: Enabling content owners to charge AI crawlers for access."
• Cloudflare: "The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals."
• TechCrunch: "Cloudflare's new policy pushes AI companies to pay for publishers' content" (July 1, 2026).
• TechCrunch: "Gemini Spark, Google's agentic assistant, is now available on Mac" (July 1, 2026).
• Forbes: "Bots Now Outnumber Humans Online And The Internet Was Never Built For This" (June 4, 2026).

Is the bot reading your site right now training a model, or booking a customer? Most site owners have no idea, and until this month, most had no way to tell the two apart.