A common issue: enabling layered navigation on a shop with many products, will produce an exponential amount of unique URLs. Besides the issue with possible SEO penalties for duplicate content, this might invoke a DDoS attack on your server.
Example: my current client has only 10K SKU's, but over a 100K daily requests from Google and Bing.
So, how to deny layered navigation for crawlers and save precious recources for real visitors?
There seems to be multiple strategies:
- Add filters to robots.txt, as described here and here
- The same, but configured in the webserver
- Add nofollow and noindex in the template files as described in this Byte blogpost
Are there more? Which is the best?