I will go straight to the point.

Recently I did several fixes in my robots.txt file, and I've added several disallows like these:

Disallow: /review/
Disallow: /catalog/product/view/

the problem is that google webmaster tools complain about duplicate meta descriptions in pages that normally shouldn't crawl:


any ideas of how can I make webmaster HTML improvements to be updated with the new robots.txt file?

further actions that I did without any result:

-updated the sitemap

-did twice fetch as Google with success

best, a

1 Answer 1


Have you tried setting the Magento 2 configs themselves to NO INDEX, NO FOLLOW? This should prevent the google robots from indexing your website and then following the website afterwards. Source: Meta noindex, follow in Magento 2.1

  • I am not sure if I understood this correctly. Setting NO INDEX, NO FOLLOW prevents google robots to index me and then if I set it up again back to INDEX FOLLOW it will update my site correctly this time? with all the excluded pages removed this time?
    – akalogiros
    Dec 21, 2018 at 7:56
  • Yes, when the google robots come to your website if set as INDEX FOLLOW they will add your website to their registry and then they will send bots within the future to get updates from your website. With NO INDEX, NO FOLLOW this will prevent the google bots from visiting your website and even adding it to their registry. You can also set it as INDEX, NO FOLLOW or NO INDEX, FOLLOW, depending on what your website requires. Sometimes just having INDEX, NO FOLLOW is useful as it gives google the search terms from your website but doesn't send any bots later to get updates. Cheers, A. Dec 21, 2018 at 9:19
  • Not a problem :) mark the answer as resolved if thats the case. Happy coding Dec 21, 2018 at 13:23
  • i will wait first some days to confirm that, but then yeah, I will mark it.
    – akalogiros
    Dec 21, 2018 at 15:08
  • Quick bump, did this end up resolving the issue? Jan 4, 2019 at 16:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.