3

I can easily set NO INDEX, FOLLOW on particular page through layout update.

But what I want is, I need to set NO INDEX, FOLLOW when the page loads after filtering from layered navigation on product listing page.

For Eg : https://test.com/recording-studio/keyboard-controllers.html?number_of_keys=37&price=-150

This is duplicate of https://test.com/recording-studio/keyboard-controllers.html. So need not to be indexed.

Any solutions for this ?

1
  • you should also consider configuring the number_of_keys query string in google console (formerly webmaster tools) so that google knows how this query affects the page.
    – paj
    Commented Jun 13, 2017 at 20:52

2 Answers 2

2

There are two options to accomplish this. First of all, you can use canonical URLs. This will tell the search engines that, although they are on a different URL, the canonical URL is the actual URL of the current page. This can be done within Magento by going to System > Configuration > Catalog > Catalog and then set the options Use Canonical Link Meta Tag For Categories and Use Canonical Link Meta Tag For Products under the tab Search Engine Optimisation to "Yes".

Second of all, you can define this in your robots.txt, which you can place in your project's root folder. This will not add a "noindex, nofollow" tag to your page, but it will tell search engines to ignore those pages.

An example of the robots.txt that I always use can be found here.

Of course you could also add a meta tag in your page, but that won't work for specific parameters (?price=xxx, etc.). For that you could use this page (also shared by Arunendra).

1
  • Could you elaborate at all on some of the things included in that robots.txt and why it would be recommended for a Magento 2 site from an SEO perspective?
    – addMitt
    Commented Jan 20, 2021 at 23:31
0

You can ignore dynamic urls using robots.txt just you need to add this line in this file you can find this file on your root folder:

Disallow: /*?*

For more see here.

5
  • I added Disallow: /*?* on robots.txt and cleared cache. run cron manually And again viewed the page with such parameters. Still no any changes Commented Apr 7, 2017 at 5:45
  • I would not suggest using this, because this will also disallow search engines to view/index the pagination of your category. So it will only index the first page (without `?p=x) and will also only not index the URLs for the products on these pages. Commented Apr 7, 2017 at 5:49
  • @sagarsapkota it will affect on search engine not on your website, it will disallow indexing from search engine please read about robots.txt
    – Arunendra
    Commented Apr 7, 2017 at 5:51
  • @ArjenMiedema why you want to index the category pagination ?? read here for more info developers.google.com/webmasters/control-crawl-index/docs/…
    – Arunendra
    Commented Apr 7, 2017 at 5:54
  • @Arunendra there is a difference between indexing and crawling. I want search engines to follow the links, so it can index the products on those pages. I don't want it to index the other pages in the pagination. That's why you should also enable the canonical URL. Commented Apr 7, 2017 at 6:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.