The Fake Googlebot Crawler That Brought My Website to Its Knees—and How I Blocked It

At first, I thought social media was taking my website down. The pattern seemed too consistent to ignore. I would publish a new blog article, share it on social media, and shortly afterward Lemon Web would become painfully slow. Pages that normally opened within seconds would take more than 10 seconds, remain pending, or fail completely. Sometimes the website recovered after a few minutes. Other times, even the CMS administrator area became almost unusable.

Social media appeared to be the obvious suspect. As it turned out, social-media crawlers were only a small part of the traffic arriving at the same time. The much bigger problem was an aggressive crawler pretending to be Googlebot.

Why Social Media Initially Looked Like the Culprit

Whenever a website link is shared on a social-media platform, the platform usually requests the page automatically to collect its title, description and featured image. This information is then used to generate the preview card shown beneath the shared link.

My server logs did show requests from social-media preview crawlers, so it was reasonable to suspect that these requests were overwhelming the website. In several cases, a social-media crawler accessed an article at around the same time that the site became slow.

However, the deeper I examined the access logs, the less convincing that explanation became.

The social-media platforms were generally making only a small number of requests. Meanwhile, hundreds of other requests were hitting blog articles, category pages and tag pages. Many arrived almost exactly one second apart, used different IP addresses and claimed to be Googlebot.

The logs showed long sequences like this:

The addresses changed sequentially while each request targeted a different page. Other legitimate crawlers and normal visitors then began receiving the same small 500 response because the web server was already struggling.

That was the turning point. Social media was not generating hundreds of requests. Something else was crawling large parts of the website while pretending to be Google.

A Googlebot User-Agent Does Not Prove It Is Google

A crawler normally identifies itself through a user-agent string included in the HTTP request. A legitimate Google crawler may identify itself as:

or:

The problem is that anyone can place those words inside a request header. A user-agent is simply a piece of text. It is not proof that the request genuinely came from Google.

Legitimate Google crawler requests can be verified through their source IP addresses or by performing forward and reverse DNS validation. A request should never be trusted simply because its user-agent contains the word Googlebot.

In my case, many of the supposed Googlebot requests came from address ranges such as:

The behaviour was very different from normal search-engine crawling. The crawler rotated through numerous addresses and requested a new tag, category or article page every second for several minutes.

Cloudflare later detected another address identifying itself as:

The address was:

Cloudflare's security-event data showed that this address belonged to the OVH SAS network in Singapore. It rapidly requested many unrelated articles, service pages and application pages using HTTP/1.0.

Cloudflare did not recognise it as a verified search-engine crawler, so the request matched my fake Googlebot rule and was blocked.

This does not mean that the hosting provider itself operated the crawler. Large cloud and hosting companies rent infrastructure to many customers. An aggressive scraper, poorly configured bot or compromised server can generate abusive traffic from a reputable provider's network without the provider actively participating in it.

How the Fake Crawler Affected My Web Server

One request to a website is normally harmless. The problem begins when every request must run the entire website application before returning a page.

An uncached blog article request may involve:

A genuine visitor might trigger that process once. An aggressive crawler can trigger it repeatedly across hundreds of different URLs.

Because the crawler requested a different page each time, an existing cache could not always serve a ready-made copy. Tag pages, category variations, query-string URLs and excluded blog articles still required the CMS, PHP and database to process each request dynamically.

As the requests accumulated, the symptoms became obvious:

The repeated error response was only around 384 bytes. This was another useful clue because it suggested that the web server was returning a small generic failure page instead of allowing the CMS to complete and render the normal page.

The issue was also intermittent. Once the crawler burst stopped and the pending PHP requests cleared, the website recovered. This explained why resource graphs sometimes appeared normal when I checked them several minutes after the incident.

The server had already recovered by the time I reached the monitoring page.

Why LiteSpeed Cache Seemed Involved

During the troubleshooting process, I enabled LiteSpeed Cache to improve the performance of the CMS. The improvement was significant, but it introduced another complication.

Caching individual blog articles caused reactions and comments to fail with an Invalid Token error. The cached HTML contained an older security token, while the reaction or comment request expected a current token.

To keep those interactive features working, I excluded individual blog articles using this rule:

This kept reactions and comments functional, but it also meant that individual article requests remained dynamic. A crawler requesting those URLs could still force PHP, the CMS and the database to process every request.

Auto Recache created an additional load spike after cache purges because the system attempted to regenerate multiple pages automatically. I therefore disabled Auto Recache completely.

LiteSpeed Cache was not proven to be the original cause of the crawler problem. However, automatic cache rebuilding made an already stressed server more vulnerable during periods of heavy bot activity.

The safer configuration became:

Caching and crawler protection were clearly two separate issues. Cache could reduce the cost of some requests, but it could not fully protect dynamic blog pages from aggressive automated traffic.

The Cloudflare Rule That Stopped the Fake Googlebot Traffic

Instead of blocking every suspicious IP address individually, I created a Cloudflare Custom Rule.

The expression was:

The action was:

In plain language, the rule means:

Cloudflare's cf.client.bot field identifies known and verified bots, including legitimate search-engine crawlers. This makes it possible to allow the real Googlebot while stopping requests that simply copy Googlebot's user-agent string.

Initially, the rule showed zero events. I wondered whether the suspicious traffic had stopped or whether the rule was matching anything at all.

It did not stay at zero for long.

The counter eventually increased to more than 190 blocked requests. Cloudflare's exported event data confirmed that the rule was actively stopping fake Googlebot traffic before those requests reached my hosting server.

One suspicious server requested dozens of unrelated pages within only a few seconds. It moved through blog articles, service pages, application pages and even shopping-related URLs while claiming to be Googlebot-Image/1.0.

That was not normal image indexing behaviour.

Adding a Challenge for Suspicious IP Ranges

The fake Googlebot rule addressed requests impersonating Google. However, some crawler traffic used ordinary browser user-agent strings instead.

To provide another layer of protection, I created a second Cloudflare rule for suspicious IP ranges that had appeared repeatedly in the server logs:

Instead of blocking those networks outright, I selected:

This was a more cautious approach. A real person visiting from one of those networks could still pass the challenge, while a basic automated crawler would usually fail or stop making requests.

The Cloudflare dashboard later showed that the rule had challenged dozens of requests.

This gave me two complementary controls:

Why I Did Not Block Entire Hosting Providers

It was tempting to block every address belonging to the networks seen in the logs. However, a complete network block can affect legitimate visitors, services or businesses using the same provider.

Large hosting and cloud providers may host thousands of unrelated customers. One abusive crawler does not mean that the entire provider is malicious.

The Cloudflare fake Googlebot rule was more precise. It did not block someone simply because they were using a particular hosting company. It acted only when the request claimed to be Googlebot but failed Cloudflare's verification.

The managed challenge rule was also safer than immediately blocking an entire range. Suspicious automated requests would be challenged, while human visitors still had a chance to continue.

This approach also protected the website when the crawler changed addresses. Blocking individual IPs would become an endless task when automated traffic rotated through hundreds of different servers.

Social Media Was the Trigger, Not the Main Cause

The timing was what made the problem so confusing.

I would publish a new article, share it on social media, and then notice the website slowing down. This made social media appear responsible.

In reality, publishing and sharing a new article can attract several different systems at once:

The social-media crawler might have been the first visible request in the log, but it represented only a small portion of the total activity.

The real damage came from aggressive automated crawlers hitting many dynamic pages while impersonating trusted search-engine bots.

Social media was therefore more of a trigger or timing indicator than the underlying cause.

What I Learned from the Incident

The biggest lesson was not to trust the most obvious coincidence.

Sharing an article on social media appeared to cause the outage because the slowdown happened soon afterward. However, the access logs showed that a much larger wave of automated traffic was arriving at the same time.

Another lesson was that a successful HTTP 200 response does not always mean a website is healthy. A page may eventually return 200 after waiting 10 or 15 seconds in a PHP processing queue. From a visitor's perspective, the website still feels unavailable.

I also learned that caching and bot protection solve different problems.

LiteSpeed Cache can reduce the amount of work required to serve pages that are already cached. Cloudflare can stop suspicious traffic before it reaches the server at all.

For aggressive crawlers targeting many different dynamic URLs, blocking or challenging the traffic at the edge is more effective than asking an overloaded CMS and PHP environment to process every request.

Most importantly, a trusted name inside a user-agent means very little. A request claiming to be Googlebot, Bingbot or another recognised crawler should not automatically be trusted without some form of verification.

Final Thoughts

Fake Googlebot traffic can be surprisingly damaging, especially on a content-heavy website where every uncached page requires PHP, a CMS, blog processing and database queries.

The crawler did not need to exploit a dramatic vulnerability or break into the server. It simply requested enough different dynamic pages quickly enough to make legitimate requests wait or fail.

That alone was enough to make the website appear unstable.

The solution was not to block Google, social media or an entire hosting provider. It was to distinguish verified crawlers from requests merely pretending to be trusted bots.

This Cloudflare rule made that possible:

Once deployed, it immediately began catching traffic that my web server should never have been processing in the first place.

The experience was frustrating, but it also offered a valuable reminder: when a website becomes slow without an obvious bandwidth, CPU or memory warning, the problem may not be normal visitor traffic.

It may be who—or what—is pretending to visit.