I'm experiencing a much higher than normal CPU load on my server since a few days mainly caused by excessive search term crawling by Google on my internal Magento search engine. This almost immediately started after I installed the "Sphinx Search" extension. I get the following kind of messages from my server:
1-0 8453 0/0/837 W 9.11 218 0 0.0 0.00 5.33 188.8.131.52 www.mywebshop.com GET /catalogsearch/result/?q=+searchterm+searchterm1+searchterm2 2-0
I've tried several solutions involving the robots.txt file all without any positive result on CPU load reduction. I could always include the following in my robots.txt:
In order to stop Google crawlers to abuse my search engine but I don't know whether this is the right way to do it and possible consequences SEO wise. Someone who has dealt with this before and can provide some advise?