Tag: AI Search

Which AI Crawlers to Block (and Which Ones Send Customers)

Picture your website as a small bakery. Most mornings, the people at the door are customers. Lately, though, there are other visitors. A food critic shows up, takes notes, and writes you into a guide that sends hungry people your way. A personal shopper comes in to grab one loaf for a customer who asked for it. A culinary school sends a student with a notebook to copy every recipe on the wall for a class you will never see. And every now and then, somebody walks in, asks to sample everything, and does it two thousand times an hour.

They are all “visitors.” Only some of them bring in business. That is the whole question behind whether you should block AI crawlers: not “AI, yes or no,” but which of these people you want in your bakery.

Why Your Logs Suddenly Look Like a Bot Convention

You are not imagining it. Cloudflare Radar put bots at 57.5% of requests for HTML pages across its network, the first time automated traffic outnumbered humans on that measure (as reported by Forbes). That is an aggregate across a huge slice of the web, not a promise about your site, but it explains why your server stats feel busier than your phone.

The anecdotes are more pointed. One small-site owner shared on X that Google read their site 43 times and sent 145 visitors, while AI bots read it 6,051 times and sent 16. Another said they blocked AI crawlers after watching ClaudeBot make more than 2,000 requests in an hour. Those are single sites, not studies. Still, they capture the frustration: a lot of reading, not much referring.

The fix is not to slam the door on everything with “bot” in its name. It is to sort the visitors.

Meet the Three Kinds of AI Bots

Most of the big AI companies now split their bots by job, and they document that split. Roughly, there are three kinds:

  1. Search crawlers (the food critics). These index pages so an AI search product can show and link to them. Blocking them can make you disappear from those answers.
  2. User-triggered fetchers (the personal shoppers). A person asks a chatbot about something, and it fetches a page right then. These aren’t broad crawls, and several vendors say robots.txt may not apply to them because a human started the request.
  3. Training crawlers (the culinary students). These collect content that may be used to train future AI models. Blocking them is an opt-out from training, and per the vendors, it doesn’t remove you from their search features.

Here is how the major names sort out, according to each company’s own documentation as of October 2026:

Keep these allowed if you want to be found:

  • Googlebot and Bingbot: classic search. Block these and you’re blocking your own SEO.
  • OAI-SearchBot: OpenAI says it is used to surface sites in ChatGPT’s search features, and opted-out sites won’t show in ChatGPT search answers.
  • ChatGPT-User: fetches pages when a ChatGPT user asks. OpenAI notes robots.txt rules may not apply, and it isn’t used to decide search inclusion.
  • Claude-SearchBot and Claude-User: Anthropic says disabling either may reduce your visibility in Claude’s search results and user-directed lookups.
  • PerplexityBot and Perplexity-User: Perplexity says PerplexityBot surfaces and links sites in its results and isn’t used to crawl content for AI foundation models. Perplexity-User generally ignores robots.txt since a person requested the fetch.
  • Applebot: powers search in Spotlight, Siri, and Safari, per Apple.

Your call (training crawlers):

  • GPTBot (OpenAI): crawls content that may be used to train its generative AI foundation models.
  • ClaudeBot (Anthropic): collects content that could contribute to model training.
  • CCBot (Common Crawl): builds an open web archive that a lot of researchers and AI developers draw from.
  • Google-Extended: not a separate crawler, just a robots.txt token. Google says it controls whether content may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI, and that it doesn’t affect Google Search inclusion or ranking.
  • Applebot-Extended: also a control token, not a crawler. It opts you out of training Apple’s foundation models while regular Applebot keeps you in Apple search.
  • meta-externalagent (Meta): Meta says it crawls for uses such as training AI models or indexing content directly.
  • Bytespider (ByteDance): appears in Cloudflare’s AI crawler analytics, but I couldn’t find a current official ByteDance page describing it, so treat it as an AI crawler you manage at the CDN.

GPTBot vs OAI-SearchBot: The Distinction That Matters Most

If you only remember one thing, make it this. GPTBot vs OAI-SearchBot is the difference between “train on my stuff” and “show my stuff to people searching in ChatGPT.” OpenAI’s docs say the two settings are independent, so you can disallow GPTBot and still allow OAI-SearchBot.

A lot of the “block AI bots” robots.txt snippets floating around list every AI-sounding name in one big Disallow block. That is the robots.txt equivalent of locking out the food critic because you were annoyed with the culinary student. If you want your business to show up when someone asks ChatGPT for a web designer in Davenport, a blanket block works against you.

A Robots.txt for ChatGPT (and Friends) That Keeps the Door Open

Here’s a sensible starting point for a small-business site that wants to stay findable in search and AI answers but opt out of model training. Your robots.txt lives at the root of your domain, right next to the XML sitemap it should point to.

robots.txt: opt out of training, stay in search

# Training crawlers and tokens: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
Disallow: /

# Everyone else (Googlebot, Bingbot, OAI-SearchBot,
# Claude-SearchBot, PerplexityBot, Applebot...): welcome
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

A few notes so this doesn’t bite you later:

  • Grouping several User-agent lines over one rule is valid, but a bot that matches a named group ignores the * group. If you also keep Disallow: /wp-admin/ style rules under *, copy them into any named group you add.
  • If you’d rather be in the training data (some brands want models to know them), delete that first block. That’s a legitimate choice, not a mistake.
  • On WordPress, the robots.txt is usually virtual. SEO plugins such as Yoast and Rank Math include a robots.txt editor, or your host can place a physical file. If you use Cloudflare, it also offers a managed robots.txt option.

robots.txt is a “Please Don’t Feed the Ducks” sign. Most visitors read it. The ducks were never going to.

Robots.txt Is a Request, Not a Lock

This is where people overestimate the file. robots.txt is a public, polite request. Well-behaved crawlers from OpenAI, Anthropic, Google, Apple, Common Crawl, and Meta say they honor it. Anything that ignores it, or simply pretends to be one of those bots by faking its user agent, walks right past. Common Crawl explicitly warns that some crawlers falsely identify themselves as CCBot.

Also note the fine print above: the user-triggered fetchers (ChatGPT-User, Perplexity-User, Meta-ExternalFetcher) may not follow robots.txt at all, because a person asked for that page. That’s usually fine. It’s one page, for one person, roughly like a browser visit.

So robots.txt handles intent. For behavior, like a crawler hammering your server, you need something that actually enforces. Same lesson as protecting your WordPress site from hackers: signs are nice, locks are better.

When a Bot Gets Greedy, Handle It at the CDN

If you’re behind a content delivery network like Cloudflare, the edge is where to deal with abusive traffic, before it ever reaches your server or your hosting bill.

Cloudflare AI Crawl Control shows which AI crawlers are hitting your site, how many requests they make, and, per Cloudflare’s docs, how often each one violates your robots.txt. You can set each crawler to Allow or Block, and blocking creates a WAF rule for you. The docs even suggest allowing crawlers that “provide value through citations, referrals,” which is the same food-critic logic.

If a bot you want to keep is simply too eager, rate-limit it instead of banning it. In Cloudflare, that’s a rate limiting rule with an expression like this:

Cloudflare rate limiting rule: expression (example)

(http.user_agent contains "ClaudeBot") or
(http.user_agent contains "Bytespider")

Pair it with a sensible threshold for a small site (say, a few dozen requests per minute) and an action like Block or Managed Challenge for the cooldown period. Pick numbers based on your own traffic, not mine.

Not on Cloudflare? Many hosts and security plugins offer rate limiting too. If your current hosting provider can’t tell you who’s eating your bandwidth, that is worth knowing on its own. Anthropic also supports the non-standard Crawl-delay line for ClaudeBot. Apple says Applebot doesn’t follow it, so don’t lean on it as your only control.

What to Expect After You Change Things

Nothing here is instant. Crawlers cache robots.txt: OpenAI says it can take about 24 hours for its search systems to adjust after a robots.txt change, Perplexity says up to 24 hours, and Meta says the same. Training opt-outs apply going forward, not retroactively, to content already collected.

And allowing search bots is not a guarantee. Being crawlable makes you eligible for AI citations, the same way being indexed makes you eligible to rank. Whether an assistant actually mentions you still depends on whether your pages answer the question clearly. Keep an eye on referrals in your analytics and in Google Search Console, and judge over months, not days.

The Short Version

  • Keep search and answer bots allowed: Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot.
  • Decide on training crawlers on purpose: GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, meta-externalagent.
  • Use robots.txt to state your preferences, and your CDN or WAF to enforce them.
  • Rate-limit a greedy bot before you ban a useful one.

Keep the Critics, Mind the Notebooks

Your bakery doesn’t need a bouncer who turns away everyone in a lanyard. It needs a door policy: critics welcome, personal shoppers fine, recipe copiers by invitation, and the two-thousand-samples-an-hour guy gets walked out.

If you’d like a second set of eyes on your robots.txt, CDN settings, or whether AI search is actually sending you anyone, that’s the kind of thing we set up for clients at Photricity. Request a quote and we’ll take a look.