I can easily set NO INDEX, FOLLOW on particular page through layout update.

But what I want is, I need to set NO INDEX, FOLLOW when the page loads after filtering from layered navigation on product listing page.

For Eg : https://test.com/recording-studio/keyboard-controllers.html?number_of_keys=37&price=-150

This is duplicate of https://test.com/recording-studio/keyboard-controllers.html. So need not to be indexed.

Any solutions for this ?

  • you should also consider configuring the number_of_keys query string in google console (formerly webmaster tools) so that google knows how this query affects the page. – paj Jun 13 '17 at 20:52

There are two options to accomplish this. First of all, you can use canonical URLs. This will tell the search engines that, although they are on a different URL, the canonical URL is the actual URL of the current page. This can be done within Magento by going to System > Configuration > Catalog > Catalog and then set the options Use Canonical Link Meta Tag For Categories and Use Canonical Link Meta Tag For Products under the tab Search Engine Optimisation to "Yes".

Second of all, you can define this in your robots.txt, which you can place in your project's root folder. This will not add a "noindex, nofollow" tag to your page, but it will tell search engines to ignore those pages.

An example of the robots.txt that I always use can be found here.

Of course you could also add a meta tag in your page, but that won't work for specific parameters (?price=xxx, etc.). For that you could use this page (also shared by Arunendra).


You can ignore dynamic urls using robots.txt just you need to add this line in this file you can find this file on your root folder:

Disallow: /*?*

For more see here.

  • I added Disallow: /*?* on robots.txt and cleared cache. run cron manually And again viewed the page with such parameters. Still no any changes – sagar sapkota Apr 7 '17 at 5:45
  • I would not suggest using this, because this will also disallow search engines to view/index the pagination of your category. So it will only index the first page (without `?p=x) and will also only not index the URLs for the products on these pages. – Arjen Miedema Apr 7 '17 at 5:49
  • @sagarsapkota it will affect on search engine not on your website, it will disallow indexing from search engine please read about robots.txt – Arunendra Apr 7 '17 at 5:51
  • @ArjenMiedema why you want to index the category pagination ?? read here for more info developers.google.com/webmasters/control-crawl-index/docs/… – Arunendra Apr 7 '17 at 5:54
  • @Arunendra there is a difference between indexing and crawling. I want search engines to follow the links, so it can index the products on those pages. I don't want it to index the other pages in the pagination. That's why you should also enable the canonical URL. – Arjen Miedema Apr 7 '17 at 6:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.