Bot or Not? How to Spot and Filter Bot Traffic in GA4

You open Google Analytics on a Monday morning and there it is: a traffic graph shaped like a hockey stick. Sessions are up 400%. For about four seconds you feel like a genius. Then you notice your new biggest market is Lanzhou, China, nobody stayed longer than zero seconds, and every one of these “visitors” showed up through Direct with a landing page of (not set).

Congratulations, you have bot traffic in GA4. You are in good company. It is one of the most common complaints in analytics circles this year, and the good news is that it is usually harmless to your site and very fixable in your reports.

Think of your analytics as a guest book at a wedding reception. Most signatures belong to real guests. But somebody left the door open, and now a busload of mannequins has been wheeled through, each one “signing” the book on the way past. They didn’t eat the cake. They didn’t dance. They just make the headcount look amazing. Your job isn’t to panic about the mannequins. It’s to learn what a mannequin signature looks like, stop counting them, and maybe put a bouncer at the door.

What GA4 Already Filters (and What It Doesn’t)

First, some credit where it’s due. According to Google’s known bot-traffic exclusion documentation, GA4 automatically excludes traffic from known bots and spiders, using Google’s own research plus the IAB’s International Spiders and Bots List. You can’t turn it off, and you can’t see how much it excluded.

The key word is known. Polite crawlers that identify themselves get caught. Bots that run a real headless browser, execute your Google Analytics tag, and pretend to be Chrome on Windows walk right past the velvet rope. So do spammers who fire fake hits straight at GA without ever touching your site.

The Telltale Signs of Bot Traffic

Mannequins are easy to spot once you know what to look for. Here are the fingerprints showing up most often in 2026:

  • Sudden Direct or Unassigned spikes. A big jump in Direct (or Unassigned) sessions that starts and stops abruptly, often with no matching bump in leads or sales.
  • Landing page “(not set)”. The session started, but no proper page_view with a page location came along. Real browsers rarely do that in bulk.
  • Weird cities. Lanzhou and Singapore have been the headline acts; WIRED reported that traffic traced back to cloud networks rather than actual residents. Data-center towns like Boardman, Oregon and Ashburn, Virginia (home to huge AWS footprints) are classic suspects too.
  • Zero engagement. Engagement rate near 0%, average engagement time measured in fractions of a second, and one page per session.
  • Suspicious sameness. 99% desktop, one browser, one operating system, one screen resolution. Humans are messier than that.
  • Referral spam. Referrals from domains you’ve never heard of, sometimes with a “visit my SEO service” vibe, that never seem to send anyone who actually reads anything.

Search Engine Watch described a September 2026 wave where Singapore sessions jumped ninefold in a day with a 0% engagement rate and about half a second per session. Some newer bots are getting better at faking human-looking behavior, so don’t rely on one signal. Look for a cluster of them.

Investigate With an Exploration

Standard reports will show you the spike. An Exploration will show you who’s behind it. Go to Explore, start a Free form exploration, and build a table like this:

Exploration recipe: "Who are these people?"

Rows:     Session default channel group
          City
          Landing page + query string
Columns:  Device category
Values:   Sessions
          Engaged sessions
          Average engagement time per session
Filter:   Session default channel group exactly matches Direct
Date:     The spike, plus a normal week before it

Then swap in Browser, Operating system, Screen resolution, and Hostname as rows. If one combination accounts for most of the spike, and its engagement is basically zero, you’ve found your mannequins. Hostname is especially useful: if hits are arriving for a domain that isn’t yours, that’s spam sent straight to your measurement ID rather than visits to your site.

Hide the Noise in Your Reports

Once you know the pattern, you have a few non-destructive ways to look past it:

  • Comparisons. In standard reports, add a comparison that excludes the suspicious city or country. Great for a quick “what does reality look like?” check.
  • Segments. In Explorations, build a session segment that excludes the bot fingerprint (say, City is Lanzhou, or Screen resolution equals that one suspicious value with zero engagement). Reuse it any time you analyze.
  • Report filters. Google itself recommends report filters when you want to hide data without permanently removing it.

These views are reversible, which matters. If you exclude all of Singapore and you actually have customers there, you can simply undo it.

Data Filters: Powerful, Permanent, and Picky

GA4 also has data filters, which stop matching events from being processed at all. Here’s the fine print you need to know before reaching for them:

  • They only work going forward. Data filters don’t affect historical data. GA4 can’t retroactively delete last week’s bot spike.
  • Exclusions are permanent. Excluded data is never processed and won’t show up in GA or BigQuery later. Always run a filter in Testing state first.
  • The menu is short. The filter types are Developer traffic, Internal traffic (by IP address), and, more recently, Web hostname traffic. There’s no “exclude Lanzhou” or “exclude screen resolution” data filter.

Internal and developer filters are still worth setting up, because your own team’s visits are a different flavor of noise. Define your office IPs under your web data stream’s tag settings (Define internal traffic), then activate the Internal traffic filter under Admin → Data collection and modification → Data filters. If you see spam arriving for strange hostnames, a hostname filter that includes only your real domain(s) is the best defense GA4 offers against fake hits.

What About Unwanted Referrals?

GA4’s List unwanted referrals setting (in your web stream’s tag settings) sounds like a spam filter, but it isn’t. It tells GA4 not to credit certain domains as referrers, which is meant for things like payment processors bouncing shoppers back to your checkout. It doesn’t delete the sessions. Use it for legitimate third-party hops, and use segments or report filters for spam.

Check the Door, Not Just the Guest Book

GA4 only sees visitors that run your tag. To see what’s really knocking, cross-check with your CDN or server. If your site sits behind Cloudflare, its traffic and security analytics show requests by country, ASN, and user agent, and paid plans add Bot Analytics with bot scores. Cloudflare’s CDN layer is also where you can actually block or challenge bad traffic with a rule, before it ever reaches your site:

# Cloudflare custom rule (expression) → Action: Managed Challenge
(ip.src.country eq "SG" and not cf.client.bot)

Adjust that to whatever your Exploration revealed, and only challenge regions you don’t actually serve. Your web host‘s raw access logs are the other source of truth: a burst of requests from one cloud provider’s IP range with one user agent tells you more than any GA dashboard can.

AI crawlers are a related but separate headache. Most of them don’t run JavaScript, so they rarely show up in GA4 at all, but they do show up in your logs and bandwidth bill. I wrote a separate guide on which AI crawlers to block with crawl rules and robots.txt.

Common Pitfalls

  • Excluding whole countries forever. Fine as a report view, risky as a habit if you sell internationally.
  • Skipping the Testing state. A sloppy data filter can permanently delete real data. Test first.
  • Blaming SEO. Bot hits in GA4 don’t hurt your rankings. Keep an eye on Google Search Console for real search performance.
  • Assuming it’s always bots. A broken tag, consent banner change, or slow-firing session_start can also produce Unassigned or (not set) data. Check your setup before you blame the robots.
  • Ignoring your forms. If bots are filling out your contact form too, that’s a security problem, not just an analytics one.

Count the Guests, Not the Mannequins

Bot traffic in GA4 is annoying, but it doesn’t have to ruin your data. Recognize the fingerprints, isolate them with an Exploration, hide them with segments and comparisons, use data filters carefully for what they can handle, and stop the worst offenders at the door with Cloudflare or your server. Then annotate the spike so future-you doesn’t spend another Monday morning wondering if you went viral in Lanzhou.

Planning a new site and want it built with clean analytics, sensible filters, and bot protection from day one? Request a custom web design quote and let’s make sure your numbers tell the truth from launch.

Which AI Crawlers to Block (and Which Ones Send Customers)

Picture your website as a small bakery. Most mornings, the people at the door are customers. Lately, though, there are other visitors. A food critic shows up, takes notes, and writes you into a guide that sends hungry people your way. A personal shopper comes in to grab one loaf for a customer who asked for it. A culinary school sends a student with a notebook to copy every recipe on the wall for a class you will never see. And every now and then, somebody walks in, asks to sample everything, and does it two thousand times an hour.

They are all “visitors.” Only some of them bring in business. That is the whole question behind whether you should block AI crawlers: not “AI, yes or no,” but which of these people you want in your bakery.

Why Your Logs Suddenly Look Like a Bot Convention

You are not imagining it. Cloudflare Radar put bots at 57.5% of requests for HTML pages across its network, the first time automated traffic outnumbered humans on that measure (as reported by Forbes). That is an aggregate across a huge slice of the web, not a promise about your site, but it explains why your server stats feel busier than your phone.

The anecdotes are more pointed. One small-site owner shared on X that Google read their site 43 times and sent 145 visitors, while AI bots read it 6,051 times and sent 16. Another said they blocked AI crawlers after watching ClaudeBot make more than 2,000 requests in an hour. Those are single sites, not studies. Still, they capture the frustration: a lot of reading, not much referring.

The fix is not to slam the door on everything with “bot” in its name. It is to sort the visitors.

Meet the Three Kinds of AI Bots

Most of the big AI companies now split their bots by job, and they document that split. Roughly, there are three kinds:

  1. Search crawlers (the food critics). These index pages so an AI search product can show and link to them. Blocking them can make you disappear from those answers.
  2. User-triggered fetchers (the personal shoppers). A person asks a chatbot about something, and it fetches a page right then. These aren’t broad crawls, and several vendors say robots.txt may not apply to them because a human started the request.
  3. Training crawlers (the culinary students). These collect content that may be used to train future AI models. Blocking them is an opt-out from training, and per the vendors, it doesn’t remove you from their search features.

Here is how the major names sort out, according to each company’s own documentation as of October 2026:

Keep these allowed if you want to be found:

  • Googlebot and Bingbot: classic search. Block these and you’re blocking your own SEO.
  • OAI-SearchBot: OpenAI says it is used to surface sites in ChatGPT’s search features, and opted-out sites won’t show in ChatGPT search answers.
  • ChatGPT-User: fetches pages when a ChatGPT user asks. OpenAI notes robots.txt rules may not apply, and it isn’t used to decide search inclusion.
  • Claude-SearchBot and Claude-User: Anthropic says disabling either may reduce your visibility in Claude’s search results and user-directed lookups.
  • PerplexityBot and Perplexity-User: Perplexity says PerplexityBot surfaces and links sites in its results and isn’t used to crawl content for AI foundation models. Perplexity-User generally ignores robots.txt since a person requested the fetch.
  • Applebot: powers search in Spotlight, Siri, and Safari, per Apple.

Your call (training crawlers):

  • GPTBot (OpenAI): crawls content that may be used to train its generative AI foundation models.
  • ClaudeBot (Anthropic): collects content that could contribute to model training.
  • CCBot (Common Crawl): builds an open web archive that a lot of researchers and AI developers draw from.
  • Google-Extended: not a separate crawler, just a robots.txt token. Google says it controls whether content may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI, and that it doesn’t affect Google Search inclusion or ranking.
  • Applebot-Extended: also a control token, not a crawler. It opts you out of training Apple’s foundation models while regular Applebot keeps you in Apple search.
  • meta-externalagent (Meta): Meta says it crawls for uses such as training AI models or indexing content directly.
  • Bytespider (ByteDance): appears in Cloudflare’s AI crawler analytics, but I couldn’t find a current official ByteDance page describing it, so treat it as an AI crawler you manage at the CDN.

GPTBot vs OAI-SearchBot: The Distinction That Matters Most

If you only remember one thing, make it this. GPTBot vs OAI-SearchBot is the difference between “train on my stuff” and “show my stuff to people searching in ChatGPT.” OpenAI’s docs say the two settings are independent, so you can disallow GPTBot and still allow OAI-SearchBot.

A lot of the “block AI bots” robots.txt snippets floating around list every AI-sounding name in one big Disallow block. That is the robots.txt equivalent of locking out the food critic because you were annoyed with the culinary student. If you want your business to show up when someone asks ChatGPT for a web designer in Davenport, a blanket block works against you.

A Robots.txt for ChatGPT (and Friends) That Keeps the Door Open

Here’s a sensible starting point for a small-business site that wants to stay findable in search and AI answers but opt out of model training. Your robots.txt lives at the root of your domain, right next to the XML sitemap it should point to.

robots.txt: opt out of training, stay in search

# Training crawlers and tokens: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
Disallow: /

# Everyone else (Googlebot, Bingbot, OAI-SearchBot,
# Claude-SearchBot, PerplexityBot, Applebot...): welcome
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

A few notes so this doesn’t bite you later:

  • Grouping several User-agent lines over one rule is valid, but a bot that matches a named group ignores the * group. If you also keep Disallow: /wp-admin/ style rules under *, copy them into any named group you add.
  • If you’d rather be in the training data (some brands want models to know them), delete that first block. That’s a legitimate choice, not a mistake.
  • On WordPress, the robots.txt is usually virtual. SEO plugins such as Yoast and Rank Math include a robots.txt editor, or your host can place a physical file. If you use Cloudflare, it also offers a managed robots.txt option.

robots.txt is a “Please Don’t Feed the Ducks” sign. Most visitors read it. The ducks were never going to.

Robots.txt Is a Request, Not a Lock

This is where people overestimate the file. robots.txt is a public, polite request. Well-behaved crawlers from OpenAI, Anthropic, Google, Apple, Common Crawl, and Meta say they honor it. Anything that ignores it, or simply pretends to be one of those bots by faking its user agent, walks right past. Common Crawl explicitly warns that some crawlers falsely identify themselves as CCBot.

Also note the fine print above: the user-triggered fetchers (ChatGPT-User, Perplexity-User, Meta-ExternalFetcher) may not follow robots.txt at all, because a person asked for that page. That’s usually fine. It’s one page, for one person, roughly like a browser visit.

So robots.txt handles intent. For behavior, like a crawler hammering your server, you need something that actually enforces. Same lesson as protecting your WordPress site from hackers: signs are nice, locks are better.

When a Bot Gets Greedy, Handle It at the CDN

If you’re behind a content delivery network like Cloudflare, the edge is where to deal with abusive traffic, before it ever reaches your server or your hosting bill.

Cloudflare AI Crawl Control shows which AI crawlers are hitting your site, how many requests they make, and, per Cloudflare’s docs, how often each one violates your robots.txt. You can set each crawler to Allow or Block, and blocking creates a WAF rule for you. The docs even suggest allowing crawlers that “provide value through citations, referrals,” which is the same food-critic logic.

If a bot you want to keep is simply too eager, rate-limit it instead of banning it. In Cloudflare, that’s a rate limiting rule with an expression like this:

Cloudflare rate limiting rule: expression (example)

(http.user_agent contains "ClaudeBot") or
(http.user_agent contains "Bytespider")

Pair it with a sensible threshold for a small site (say, a few dozen requests per minute) and an action like Block or Managed Challenge for the cooldown period. Pick numbers based on your own traffic, not mine.

Not on Cloudflare? Many hosts and security plugins offer rate limiting too. If your current hosting provider can’t tell you who’s eating your bandwidth, that is worth knowing on its own. Anthropic also supports the non-standard Crawl-delay line for ClaudeBot. Apple says Applebot doesn’t follow it, so don’t lean on it as your only control.

What to Expect After You Change Things

Nothing here is instant. Crawlers cache robots.txt: OpenAI says it can take about 24 hours for its search systems to adjust after a robots.txt change, Perplexity says up to 24 hours, and Meta says the same. Training opt-outs apply going forward, not retroactively, to content already collected.

And allowing search bots is not a guarantee. Being crawlable makes you eligible for AI citations, the same way being indexed makes you eligible to rank. Whether an assistant actually mentions you still depends on whether your pages answer the question clearly. Keep an eye on referrals in your analytics and in Google Search Console, and judge over months, not days.

The Short Version

  • Keep search and answer bots allowed: Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot.
  • Decide on training crawlers on purpose: GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, meta-externalagent.
  • Use robots.txt to state your preferences, and your CDN or WAF to enforce them.
  • Rate-limit a greedy bot before you ban a useful one.

Keep the Critics, Mind the Notebooks

Your bakery doesn’t need a bouncer who turns away everyone in a lanyard. It needs a door policy: critics welcome, personal shoppers fine, recipe copiers by invitation, and the two-thousand-samples-an-hour guy gets walked out.

If you’d like a second set of eyes on your robots.txt, CDN settings, or whether AI search is actually sending you anyone, that’s the kind of thing we set up for clients at Photricity. Request a quote and we’ll take a look.

What’s new with the web in 2023?

The web is constantly evolving, and there are always new trends and developments to keep up with. Here are some of the most exciting new things happening on the web in 2023:

The Rise of Artificial Intelligence

AI is already having a major impact on the web, and its influence is only going to grow in the years to come. AI is being used to power everything from chatbots and virtual assistants to personalized recommendations and search results. As AI becomes more sophisticated, it will have an even greater impact on the way we interact with the web.

The Continued Growth of Mobile Devices

Smartphones and tablets are now the primary way that people access the internet, and this trend is only going to continue. This means that web developers need to make sure that their websites are optimized for mobile devices. In addition, mobile devices are opening up new opportunities for businesses to reach their customers. For example, businesses can now use mobile apps to provide customer service, offer loyalty programs, and sell products.

Blockchain

Blockchain is a distributed ledger technology that is used to record transactions in a secure and transparent way. In 2023, we can expect to see more businesses using blockchain to power their online transactions, as it offers a number of advantages over traditional payment methods. Blockchain is enabling new applications, such as decentralized finance (DeFi) and non-fungible tokens (NFTs). Decentralized technology is still in its early stages, but it has the potential to revolutionize the web.

Progressive Web Applications

PWAs are a type of web app that combines the best of both worlds from websites and mobile apps. They are fast, reliable, and work offline, making them a great option for businesses that want to provide their users with a seamless experience across all devices.

Metaverse

The metaverse is a hypothesized iteration of the internet as a single, universal, and immersive virtual world that is facilitated by the use of virtual reality and augmented reality headsets. The metaverse is still in its early stages of development, but it has the potential to change the way we interact with the internet.

New web technologies are constantly being developed, and these technologies are changing the way that we use the web. For example, WebXR (the successor to the now-deprecated WebVR) allows users to experience virtual and augmented reality content on the web. WebGL is growing in popularity, a JavaScript API that enables developers to produce interactive and immersive experiences for the web. The development of new web technology has enabled many features that were previously impossible or prohibitively difficult to implement.

These are just some of the most exciting new things happening on the web in 2023. The web is a constantly evolving platform, and there are always new trends and developments to keep up with. By staying informed about the latest trends, web developers can ensure that their websites are up-to-date and relevant to their users. As technology advances, it will be interesting to see what new trends and technologies emerge in the years to come.