1

A common issue: enabling layered navigation on a shop with many products, will produce an exponential amount of unique URLs. Besides the issue with possible SEO penalties for duplicate content, this might invoke a DDoS attack on your server.

Example: my current client has only 10K SKU's, but over a 100K daily requests from Google and Bing.

So, how to deny layered navigation for crawlers and save precious recources for real visitors?

There seems to be multiple strategies:

  1. Add filters to robots.txt, as described here and here
  2. The same, but configured in the webserver
  3. Add nofollow and noindex in the template files as described in this Byte blogpost

Are there more? Which is the best?

1
  • If they all work this question is a matter of opinion imho. Nov 8, 2014 at 23:55

2 Answers 2

1

I'm no SEO expert but what I've understood from someone who is a robots.txt allows for the URL to be indexed but the content to be hidden while a nofollow on a link prevents the whole URL to be indexed.

Again, I'm no SEO expert so if anyone can confirm it that would be great

1

It is also possible

1) to hide layered navigation block from the output based on the user agents and filter our Google/Bing bots.

2) add a special variable in the url like "noind" and then in the Google webmaster tools set to noindex such pages. See https://support.google.com/webmasters/answer/6080550

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.