1

I'm experiencing a much higher than normal CPU load on my server since a few days mainly caused by excessive search term crawling by Google on my internal Magento search engine. This almost immediately started after I installed the "Sphinx Search" extension. I get the following kind of messages from my server:

1-0 8453 0/0/837 W 9.11 218 0 0.0 0.00 5.33 66.249.78.24 www.mywebshop.com GET /catalogsearch/result/?q=+searchterm+searchterm1+searchterm2 2-0

I've tried several solutions involving the robots.txt file all without any positive result on CPU load reduction. I could always include the following in my robots.txt:

Allowable Index

Allow: /*?q=

In order to stop Google crawlers to abuse my search engine but I don't know whether this is the right way to do it and possible consequences SEO wise. Someone who has dealt with this before and can provide some advise?

Thanks!

  • sphinx has some cache , isnt it? you have some cache , right? – user2857 May 12 '15 at 11:39
  • Nocache/nofollow the search results pages using local.xml to set the meta tag in the page head and rel=nofollow any links to search. Google and Yandex can quickly eat server resources if allowed into the search functions, no matter how efficient your installed search engine. Search is for your customer's convenience, Goog, et.al. can find it all the old fashioned way, a link at a time. – Fiasco Labs May 13 '15 at 0:13
1

All very good suggestions and we have come up with a solution that works very well. The challenge is to tell Google not to do something without upsetting its algorithm to much. Basically we want to slow down Google without being penalized in the end. Good points about the search functionality on the site, this is for customers and definitely not for bots. Eventually this might get you run into trouble since the threat of duplicate content waiting.

What we basically did was tell Google not to use the search functionality by adding the following code to the robots.txt file:

# Allowable Index
Disallow: /catalogsearch/
Disallow: /*?*
0

maybe you can try some rules to cool down search bots for a while:

RewriteCond %{HTTP_USER_AGENT} (Googlebot|bingbot|Yahoo) [NC]
RewriteCond %{REQUEST_URI} ^/(catalogsearch|checkout|wishlist|review)/
RewriteRule .* - [G]

Obviously there is nothing bad in telling search bot that these locations are only for users. you just send "410 - Gone", this is the correct way to talk to Google. Next time Google will know that this location is not there, and wont bother you again.

dont forget to keep your sitemap fresh.

0

We had the same problem with bots following links to search results and the layered navigation. And not just with Google, Bing and other bots are just as annoying at times.

First attempt to get these bots to behave was using robots.txt, the easy and friendly way but not very effective.

As a final attempt to get the load down we changed the server config to always send a http code 403 (Forbidden) to bots trying to access a location we don't want to be indexed or on pages with an active filter. After this change the load dropped.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.