I have a problem with my magento store. Every now and then when a bot crawls my shop, the website goes down.

I have made changes to robots.txt and .htacces. Also varnish is installed and a full page cache.

But still it doesn't help. Does anybody have some tips what I can try?


1 Answer 1


There are dozens of different bots and not all of them play nice, but you have handful of tools at your disposal:

  1. robots.txt has crawl-delay command. This makes sure that crawler waits at least specified number of seconds before making next query to the server.
  2. Google Webmaster Tool provides an option to set crawl rate. Obviously it only works for Googlebot. As far as I know Googlebot ignores crawl-delay command.
  3. Create a rule for your web server to kick out those bots who ignore previous two options. You can safely ignore vast majority of them without losing a single customer.
  • the bad bots will not follow robots.txt so that is not an option. Google bot is not the problem. Can you eleborate option 3? Thx for the response. Apr 26, 2015 at 20:17
  • You can do it for example using .htaccess deny directive (provided, that you use Apache). An example can be found here: supportcenter.verio.com/KB/questions.php?questionid=504
    – Pronto
    Apr 26, 2015 at 21:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.