High CPU load due to Google crawling search terms

Question

I'm experiencing a much higher than normal CPU load on my server since a few days mainly caused by excessive search term crawling by Google on my internal Magento search engine. This almost immediately started after I installed the "Sphinx Search" extension. I get the following kind of messages from my server:

1-0 8453 0/0/837 W 9.11 218 0 0.0 0.00 5.33 66.249.78.24 www.mywebshop.com GET /catalogsearch/result/?q=+searchterm+searchterm1+searchterm2 2-0

I've tried several solutions involving the robots.txt file all without any positive result on CPU load reduction. I could always include the following in my robots.txt:

Allowable Index

Allow: /*?q=

In order to stop Google crawlers to abuse my search engine but I don't know whether this is the right way to do it and possible consequences SEO wise. Someone who has dealt with this before and can provide some advise?

Thanks!

Answer 1

All very good suggestions and we have come up with a solution that works very well. The challenge is to tell Google not to do something without upsetting its algorithm to much. Basically we want to slow down Google without being penalized in the end. Good points about the search functionality on the site, this is for customers and definitely not for bots. Eventually this might get you run into trouble since the threat of duplicate content waiting.

What we basically did was tell Google not to use the search functionality by adding the following code to the robots.txt file:

# Allowable Index
Disallow: /catalogsearch/
Disallow: /*?*

Answer 2

maybe you can try some rules to cool down search bots for a while:

RewriteCond %{HTTP_USER_AGENT} (Googlebot|bingbot|Yahoo) [NC]
RewriteCond %{REQUEST_URI} ^/(catalogsearch|checkout|wishlist|review)/
RewriteRule .* - [G]

Obviously there is nothing bad in telling search bot that these locations are only for users. you just send "410 - Gone", this is the correct way to talk to Google. Next time Google will know that this location is not there, and wont bother you again.

dont forget to keep your sitemap fresh.

Answer 3

We had the same problem with bots following links to search results and the layered navigation. And not just with Google, Bing and other bots are just as annoying at times.

First attempt to get these bots to behave was using robots.txt, the easy and friendly way but not very effective.

As a final attempt to get the load down we changed the server config to always send a http code 403 (Forbidden) to bots trying to access a location we don't want to be indexed or on pages with an active filter. After this change the load dropped.