Which AI Crawlers to Block (and Which Ones Send Customers)

Picture your website as a small bakery. Most mornings, the people at the door are customers. Lately, though, there are other visitors. A food critic shows up, takes notes, and writes you into a guide that sends hungry people your way. A personal shopper comes in to grab one loaf for a customer who asked for it. A culinary school sends a student with a notebook to copy every recipe on the wall for a class you will never see. And every now and then, somebody walks in, asks to sample everything, and does it two thousand times an hour.

They are all “visitors.” Only some of them bring in business. That is the whole question behind whether you should block AI crawlers: not “AI, yes or no,” but which of these people you want in your bakery.

Why Your Logs Suddenly Look Like a Bot Convention

You are not imagining it. Cloudflare Radar put bots at 57.5% of requests for HTML pages across its network, the first time automated traffic outnumbered humans on that measure (as reported by Forbes). That is an aggregate across a huge slice of the web, not a promise about your site, but it explains why your server stats feel busier than your phone.

The anecdotes are more pointed. One small-site owner shared on X that Google read their site 43 times and sent 145 visitors, while AI bots read it 6,051 times and sent 16. Another said they blocked AI crawlers after watching ClaudeBot make more than 2,000 requests in an hour. Those are single sites, not studies. Still, they capture the frustration: a lot of reading, not much referring.

The fix is not to slam the door on everything with “bot” in its name. It is to sort the visitors.

Meet the Three Kinds of AI Bots

Most of the big AI companies now split their bots by job, and they document that split. Roughly, there are three kinds:

  1. Search crawlers (the food critics). These index pages so an AI search product can show and link to them. Blocking them can make you disappear from those answers.
  2. User-triggered fetchers (the personal shoppers). A person asks a chatbot about something, and it fetches a page right then. These aren’t broad crawls, and several vendors say robots.txt may not apply to them because a human started the request.
  3. Training crawlers (the culinary students). These collect content that may be used to train future AI models. Blocking them is an opt-out from training, and per the vendors, it doesn’t remove you from their search features.

Here is how the major names sort out, according to each company’s own documentation as of October 2026:

Keep these allowed if you want to be found:

  • Googlebot and Bingbot: classic search. Block these and you’re blocking your own SEO.
  • OAI-SearchBot: OpenAI says it is used to surface sites in ChatGPT’s search features, and opted-out sites won’t show in ChatGPT search answers.
  • ChatGPT-User: fetches pages when a ChatGPT user asks. OpenAI notes robots.txt rules may not apply, and it isn’t used to decide search inclusion.
  • Claude-SearchBot and Claude-User: Anthropic says disabling either may reduce your visibility in Claude’s search results and user-directed lookups.
  • PerplexityBot and Perplexity-User: Perplexity says PerplexityBot surfaces and links sites in its results and isn’t used to crawl content for AI foundation models. Perplexity-User generally ignores robots.txt since a person requested the fetch.
  • Applebot: powers search in Spotlight, Siri, and Safari, per Apple.

Your call (training crawlers):

  • GPTBot (OpenAI): crawls content that may be used to train its generative AI foundation models.
  • ClaudeBot (Anthropic): collects content that could contribute to model training.
  • CCBot (Common Crawl): builds an open web archive that a lot of researchers and AI developers draw from.
  • Google-Extended: not a separate crawler, just a robots.txt token. Google says it controls whether content may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI, and that it doesn’t affect Google Search inclusion or ranking.
  • Applebot-Extended: also a control token, not a crawler. It opts you out of training Apple’s foundation models while regular Applebot keeps you in Apple search.
  • meta-externalagent (Meta): Meta says it crawls for uses such as training AI models or indexing content directly.
  • Bytespider (ByteDance): appears in Cloudflare’s AI crawler analytics, but I couldn’t find a current official ByteDance page describing it, so treat it as an AI crawler you manage at the CDN.

GPTBot vs OAI-SearchBot: The Distinction That Matters Most

If you only remember one thing, make it this. GPTBot vs OAI-SearchBot is the difference between “train on my stuff” and “show my stuff to people searching in ChatGPT.” OpenAI’s docs say the two settings are independent, so you can disallow GPTBot and still allow OAI-SearchBot.

A lot of the “block AI bots” robots.txt snippets floating around list every AI-sounding name in one big Disallow block. That is the robots.txt equivalent of locking out the food critic because you were annoyed with the culinary student. If you want your business to show up when someone asks ChatGPT for a web designer in Davenport, a blanket block works against you.

A Robots.txt for ChatGPT (and Friends) That Keeps the Door Open

Here’s a sensible starting point for a small-business site that wants to stay findable in search and AI answers but opt out of model training. Your robots.txt lives at the root of your domain, right next to the XML sitemap it should point to.

robots.txt: opt out of training, stay in search

# Training crawlers and tokens: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
Disallow: /

# Everyone else (Googlebot, Bingbot, OAI-SearchBot,
# Claude-SearchBot, PerplexityBot, Applebot...): welcome
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

A few notes so this doesn’t bite you later:

  • Grouping several User-agent lines over one rule is valid, but a bot that matches a named group ignores the * group. If you also keep Disallow: /wp-admin/ style rules under *, copy them into any named group you add.
  • If you’d rather be in the training data (some brands want models to know them), delete that first block. That’s a legitimate choice, not a mistake.
  • On WordPress, the robots.txt is usually virtual. SEO plugins such as Yoast and Rank Math include a robots.txt editor, or your host can place a physical file. If you use Cloudflare, it also offers a managed robots.txt option.

robots.txt is a “Please Don’t Feed the Ducks” sign. Most visitors read it. The ducks were never going to.

Robots.txt Is a Request, Not a Lock

This is where people overestimate the file. robots.txt is a public, polite request. Well-behaved crawlers from OpenAI, Anthropic, Google, Apple, Common Crawl, and Meta say they honor it. Anything that ignores it, or simply pretends to be one of those bots by faking its user agent, walks right past. Common Crawl explicitly warns that some crawlers falsely identify themselves as CCBot.

Also note the fine print above: the user-triggered fetchers (ChatGPT-User, Perplexity-User, Meta-ExternalFetcher) may not follow robots.txt at all, because a person asked for that page. That’s usually fine. It’s one page, for one person, roughly like a browser visit.

So robots.txt handles intent. For behavior, like a crawler hammering your server, you need something that actually enforces. Same lesson as protecting your WordPress site from hackers: signs are nice, locks are better.

When a Bot Gets Greedy, Handle It at the CDN

If you’re behind a content delivery network like Cloudflare, the edge is where to deal with abusive traffic, before it ever reaches your server or your hosting bill.

Cloudflare AI Crawl Control shows which AI crawlers are hitting your site, how many requests they make, and, per Cloudflare’s docs, how often each one violates your robots.txt. You can set each crawler to Allow or Block, and blocking creates a WAF rule for you. The docs even suggest allowing crawlers that “provide value through citations, referrals,” which is the same food-critic logic.

If a bot you want to keep is simply too eager, rate-limit it instead of banning it. In Cloudflare, that’s a rate limiting rule with an expression like this:

Cloudflare rate limiting rule: expression (example)

(http.user_agent contains "ClaudeBot") or
(http.user_agent contains "Bytespider")

Pair it with a sensible threshold for a small site (say, a few dozen requests per minute) and an action like Block or Managed Challenge for the cooldown period. Pick numbers based on your own traffic, not mine.

Not on Cloudflare? Many hosts and security plugins offer rate limiting too. If your current hosting provider can’t tell you who’s eating your bandwidth, that is worth knowing on its own. Anthropic also supports the non-standard Crawl-delay line for ClaudeBot. Apple says Applebot doesn’t follow it, so don’t lean on it as your only control.

What to Expect After You Change Things

Nothing here is instant. Crawlers cache robots.txt: OpenAI says it can take about 24 hours for its search systems to adjust after a robots.txt change, Perplexity says up to 24 hours, and Meta says the same. Training opt-outs apply going forward, not retroactively, to content already collected.

And allowing search bots is not a guarantee. Being crawlable makes you eligible for AI citations, the same way being indexed makes you eligible to rank. Whether an assistant actually mentions you still depends on whether your pages answer the question clearly. Keep an eye on referrals in your analytics and in Google Search Console, and judge over months, not days.

The Short Version

  • Keep search and answer bots allowed: Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot.
  • Decide on training crawlers on purpose: GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, meta-externalagent.
  • Use robots.txt to state your preferences, and your CDN or WAF to enforce them.
  • Rate-limit a greedy bot before you ban a useful one.

Keep the Critics, Mind the Notebooks

Your bakery doesn’t need a bouncer who turns away everyone in a lanyard. It needs a door policy: critics welcome, personal shoppers fine, recipe copiers by invitation, and the two-thousand-samples-an-hour guy gets walked out.

If you’d like a second set of eyes on your robots.txt, CDN settings, or whether AI search is actually sending you anyone, that’s the kind of thing we set up for clients at Photricity. Request a quote and we’ll take a look.

Schema Markup: Enrich Your Search Engine Appearance With Structured Data

Web designers and SEO professionals are constantly seeking ways to optimize websites for search engines. One powerful technique that has gained significant traction is the implementation of schema markup and structured data. These innovative concepts not only enhance the way search engines understand and index web content but also improve the overall user experience.

Imagine if search engines had a better understanding of your website’s content, allowing them to display rich snippets, knowledge panels, and other visually appealing elements in search results. This is where schema markup comes into play. It provides search engines with structured information, making it easier for them to interpret and present your web content accurately.

What is Schema Markup and Structured Data?

Schema markup is a specific vocabulary or code language that allows webmasters to provide structured information about their web content. It helps search engines understand the meaning and context of the data on a webpage. With it web designers can classify and organize their content into specific categories or types, making it easier for search engines to interpret and present relevant information to users.

Structured data refers to the actual content that is marked up using schema markup. It provides additional context and details about various elements on a webpage, such as products, articles, events, recipes, and more. This data helps web designers supply search engines with specific information about their content, such as titles, descriptions, ratings, prices, and dates, enriching search results with relevant and visually appealing elements.

Implementing structured data brings several advantages. First, it enhances visibility in search results by enabling search engines to display rich snippets, knowledge panels, and other visually appealing elements. This improved visibility can significantly increase click-through rates and drive more targeted traffic to your website. Schema markup also contributes to a better user experience by providing users with accurate and detailed information directly on the search engine results page. Users can quickly find what they’re looking for without having to navigate through multiple web pages.

Types of Schema Markup

Schema markup offers a wide range of types that cater to various formats of web content. Let’s explore some of the most commonly used types:

  1. Organization: This type allows you to provide detailed information about your organization, such as its name, logo, contact details, social media profiles, and more. Implementing organization structured data helps search engines accurately represent your business in search results and knowledge panels.
  2. Local Business: If you have a physical location or operate locally, local business schema markup is crucial. It enables you to provide specific details about your business address, opening hours, phone number, and other relevant information. This helps local searchers find your business more easily.
  3. Product: Product markup allows you to showcase your products with rich snippets in search results. You can provide product details like name, description, price, availability, and even customer reviews. Implementing product schema markup can increase the visibility of your products and improve their chances of appearing in product-related search results.
  4. Article: Article schema markup is ideal for publishers and bloggers. It allows you to provide structured data about your articles, including the headline, author, publication date, and a brief summary.
  5. Review: Review markup helps you display customer reviews and ratings in search results. You can provide information such as the reviewer’s name, rating, review text, and even the product or service being reviewed. This type of data adds credibility to your business and can influence user decisions.

Each type of schema markup comes with its own set of properties and attributes. For example, product markup includes properties like brand, SKU, aggregateRating, and offers, while article schema markup uses properties such as headline, author, datePublished, and articleBody. Understanding these properties and attributes is essential for correctly implementing Schema Markup and ensuring that search engines interpret your data accurately.

Let’s consider some practical examples of structured data implementation. For a business website, you can apply organization markup to provide your company’s name, logo, and contact information. If you offer products, incorporating product schema enables you to display product details like name, price, and availability directly in search results. If you publish articles, utilizing article markup allows search engines to present your article’s headline, author, and publication date in a visually appealing manner. And if your website includes customer reviews, you can implement review markup to allow search engines to display star ratings and review snippets, boosting the credibility of your business.

Implementing Schema Markup

There are several methods you can use to implement schema markup on your website. Let’s explore two common approaches:

  1. Manual Coding: This method involves manually adding structured data to your HTML code. You can insert the appropriate schema markup tags and attributes directly into the relevant sections of your HTML markup. While this method offers more control and customization options, it requires a solid understanding of schema markup syntax and HTML coding.
  2. Structured Data Generators and Plugins: Free schema generators and SEO plugins (like Yoast SEO or Rank Math for WordPress) simplify the process of adding schema markup. These tools offer user-friendly interfaces where you can input your data and generate the necessary code. This method is especially helpful if you’re new to schema markup or prefer a more guided and automated approach.

Here’s an example of structured data using schema markup for a business website:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "LocalBusiness",
  "name": "Example Business",
  "address": {
    "@type": "PostalAddress",
    "streetAddress": "123 Main Street",
    "addressLocality": "Cityville",
    "addressRegion": "State",
    "postalCode": "12345",
    "addressCountry": "US"
  },
  "telephone": "+1-555-123-4567",
  "url": "https://www.example.com",
  "openingHours": "Mo-Fr 09:00-18:00",
  "image": "https://www.example.com/logo.png",
  "priceRange": "$$",
  "description": "Example Business is a leading provider of quality products/services in the industry."
}
</script>

In this example, we’re using the LocalBusiness type to represent a local business entity. The structured data includes the business name, address, telephone number, website URL, opening hours, image/logo, price range, and a brief description.

After implementing your schema markup, it’s essential to validate your markup to ensure it’s correctly structured and error-free. Google retired its old Structured Data Testing Tool, so today you’ll want Google’s Rich Results Test (or the general-purpose Schema Markup Validator at validator.schema.org). Follow these steps to validate your code:

  1. Open the Rich Results Test in your web browser.
  2. Enter the URL of the page containing the schema markup or paste the markup code directly into the tool.
  3. Click on the “Test URL” (or “Test Code”) button to analyze the markup.
  4. The tool will provide feedback on the validity of the markup, highlighting any errors or warnings that need attention.

The Future of Structured Data

The landscape of structured data continues to evolve, with exciting advancements and emerging trends shaping the future of these technologies:

  • Expanded Schema Types: As the demand for structured information grows, Schema.org continues to expand its vocabulary with new schema types. These additions cater to specific industries, niches, and emerging technologies, allowing website owners to provide more precise and detailed information to search engines.
  • Voice-Activated Assistants: The proliferation of voice-activated assistants presents new opportunities for schema markup. As voice search becomes increasingly prevalent, schema markup will play a vital role in optimizing content for voice queries. Future developments may include schema enhancements specifically tailored to voice search, enabling websites to provide even more contextually relevant information.
  • Artificial Intelligence (AI) Integration: As search engines become more sophisticated with AI and machine learning, schema markup could play a crucial role in helping algorithms better understand and interpret web content. AI-driven algorithms can leverage structured data to improve search relevance, enhance user experiences, and deliver more accurate and personalized results.

Schema markup and structured data offer significant benefits for web designers and SEO professionals. It empowers websites to provide search engines with structured information about their content, leading to improved visibility, higher click-through rates, and enhanced user experiences. Understanding different types of schema markup, implementing it correctly, and staying updated with the latest developments are crucial for harnessing the full potential of structured data. Embrace the power of schema markup and structured data to optimize your website for search engines and deliver exceptional user experiences.