How to detect and Modify robots.txt inside Magento

Question

How do I modify and ensure that the magento site has robots.txt please?

I am checking if the site has robots.txt as it is being flagged by our SEo tool . see screenshot - http://prntscr.com/k07bvl

Thanks!

Answer 1

Configure Magento for generating robots.txt To generate a robots.txt file, use the following steps:

Login in your Magento admin backend Select

1. Store -> Configuration
2. Select General -> Design
3. Select the dropdown Search Engine Robots.
4. Select the default storefront or one you want to create a robots.txt

For Default Robots, use one of the available options:

INDEX, FOLLOW: Tell crawlers to index the site and to keep doing this periodically.

NOINDEX, FOLLOW: Tell crawlers not to index the site but to check for changes of this policy periodically.

INDEX, NOFOLLOW: Tell crawlers to index the shop just once and don’t check for changes periodically.

NOINDEX, NOFOLLOW: Tell crawlers not to index the shop and don’t check for changes periodically.

After this is done, click the Reset to Default button to add the default robots.txt instructions to the custom instructions field.

You can now add your custom instructions to the default. After editing, click the Save Config button to save your robots.txt file to disk.