How do I modify and ensure that the magento site has robots.txt please?

I am checking if the site has robots.txt as it is being flagged by our SEo tool . see screenshot - http://prntscr.com/k07bvl



1 Answer 1


Configure Magento for generating robots.txt To generate a robots.txt file, use the following steps:

Login in your Magento admin backend Select

  1. Store -> Configuration
  2. Select General -> Design
  3. Select the dropdown Search Engine Robots.
  4. Select the default storefront or one you want to create a robots.txt

For Default Robots, use one of the available options:

INDEX, FOLLOW: Tell crawlers to index the site and to keep doing this periodically.

NOINDEX, FOLLOW: Tell crawlers not to index the site but to check for changes of this policy periodically.

INDEX, NOFOLLOW: Tell crawlers to index the shop just once and don’t check for changes periodically.

NOINDEX, NOFOLLOW: Tell crawlers not to index the shop and don’t check for changes periodically.

After this is done, click the Reset to Default button to add the default robots.txt instructions to the custom instructions field.

You can now add your custom instructions to the default. After editing, click the Save Config button to save your robots.txt file to disk.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.