0

I took over a magento site running on nginx. I looked and saw that currently there are ~100 "online" guests, all of them are bingbot. There are a lot of suggestions for how to properly block bots, referers, and user-agents (such as this one), but none of them really explain their code and how it does what they say it does. I am wondering if anyone could suggest code for bot/user-agent/referer blocking in nginx, and explain why that code works.

1

The guest count comes from sessions. Bots do not send cookies, so each request will be a be session. The Redis Session module has code to combat this issue and I highly recommend switching to Redis session storage backend.

The second issue is that most likely the bots are crawling your layered navigation. You need to determine what the layered navigation URL components are and disallow them in your robots.txt. Disallow follows standard glob syntax.

Layered navigation are the filters on category pages that slice the product collection of that category based on things like sub category, price, color etc. Since each site has different filters setup and some have extension that alter the generated URLs, this has to be customized.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.